Contao: Mark a folder as not to synchronize even if there are no rights to read nor to write

[schnudd31do3]

In an earlier version, you could specify the folders and files to be excluded in an input field, separated by commas. In the current version of the contao file system I integrated a CIFS share that can only be read. Therefore when I click "Don't sync", I get this error message: Internal Server Error: Failed to touch ".../.nosync".

I also have a CIFS share which is a distributed file system. For synchronizing on the Windows side it contains a none radable Folder by ls -la:
d????????? ? ? ? ? ? MyPrivateName
If I want to edit it in contao for not-synchronizing, I get this error message:
Forbidden: What is the problem?, File or folder ".../MyPrivateName" is not mounted or cannot be found.

[asaage]

You can create an (empty) file with filename .nosync in a folder manually to exclude it from synchronisation.
But if the folder in no-read - what's the point of mounting it into tl_files?

[schnudd31do3]

1. The .nosync folder is readable but not writable. How can contao place this file in it? It has no sence to place it manually because I need automatic processing.
2. The none readable folder is listed in contao and I can not exclude it from synchronizing. Without excluding it I cannot use synchronize function.

[m-vo]

. Therefore when I click "Don't sync", I get this error message: Internal Server Error: Failed to touch ".../.nosync".

Contao adds an empty .nosync file as a marker that this folder should not be used when creating the sync diff. You'll have to temporarily mount your share rw to do so. As @asaage pointed out you can also create the file yourself.

[asaage]

As far as i know the .nosync file is the only indicator to not sync a folder (including subfolders)

[schnudd31do3]

@m-vo: There is no point in doing this manually. With us, all processes are controlled by script. In addition, manual maintenance is not process-reliable. Folders were intentionally mounted as read-only. It wouldn't make sense if it were possible to write this file (.nosync) neither manually nor automatically.
Have you read this from me:
"2. The none readable folder is listed in contao and I can not exclude it from synchronizing. Without excluding it I cannot use synchronize function." How to avoid this?
@asaage, m-vo: Wouldn't it be possible to go back to the previous version's approach?

[m-vo]

I don't get your point. Why don't you want to use the DBAFS with your (read-only) share?

[schnudd31do3]

As described, there are two different problems with contao. These two problems are independent of each other.

The first problem is that folders I not want to be synchronized cannot be excluded from the synchronization because contao wants to write the .nosync file to readonly folders. For security reasons we dont want applications or persons to write to archive folders or modify them in any way.

The second problem is that the DFS have unreadable folders (see d????????? ? ? ? ? ? MyPrivateName) for internaly usage which are given. These folders are unreadable due to empty file system information and not due to incorrect permissions. This specific folders prevent from being synchronizing by contao because they cannot be excluded from synchronization, see "Forbidden: What is the problem?, File or folder ".../MyPrivateName" is not mounted or cannot be found.".

[m-vo]

The first problem is that folders I not want to be synchronized cannot be excluded from the synchronization because contao wants to write the .nosync file [...]

That's just how Contao works? :shrug:

The second problem is that the DFS have unreadable folders (see d????????? ? ? ? ? ? MyPrivateName) for internaly usage which are given. These folders are unreadable due to empty file system information and not due to incorrect permissions.

If this can be fixed a PR is welcome I think. Can you point to where this occurs in the code?

[fritzmg]

How can contao place this file in it?

@schnudd31do3 I think @asaage 's point was, that you should simply put the .nosync file there yourself. Then there should not be a problem, or is there?

[schnudd31do3]

@m-vo: "That's just how Contao works? 🤷" It is not a good idea to write on archives. The point of archives is that they remain unchanged

@fritzmg: I've known this option for a long time. From my point of view it is a quick & dirty solution. Please take a look at the previous posts from me. It says why such a procedure does not make sense.

I also found out that contao wants to create in the unreadable DFS folder other folders such as a CSS folder, but it cannot because no file system information is available for these DFS folders.

The solution for both problems could, from my point of view, be that it was solved by only comparing exclusion lists with directory contents. Contao would then no longer try to access the problematic DFS folder and not write to archive folders (for security policies) by not writing this .nosync file into.

[fritzmg]

From my point of view it is a quick & dirty solution.

From our point of view it is the correct and intended solution ;). Folders are marked as not being synchronized through a .nosync file, public folders are marked through a .public file.

However, I think it would make sense to be able to configure folders for either of these options through a core bundle configuration option as well.

I also found out that contao wants to create in the unreadable DFS folder other folders such as a CSS folder

Not sure what you mean there. Dynamically generated JS and CSS assets are stored within assets/js/ and assets/css/ respectively, along with assets/images/ for dynamically generated images.

[asaage]

However, I think it would make sense to be able to configure folders for either of these options through a core bundle configuration option as well.

That is probably not a bad idea.

@schnudd31do3 for the time being i suggest you could still create the folders like such:

• project.root
  
  
  
  • files
  
  
  • NoSyncFolder (with .nosync file inside)
    
    
    
    
    
    • mounted read-only CIFS-share
    
    
    
  
  

[schnudd31do3]

Your statements seem to tell me that everything is OK the way it is and it will stay that way. Your proposed "solutions" do not match our understanding of IT security. Also not about the integrity of approved version statuses and also not about automated work processes. Nor do they solve the problem if contao wants to access folders with missing file information. If Contao has solved the two bugs I mentioned in my first post, we can use Contao in our company.

[fritzmg]

Which bugs do you mean? Also please do not report different kinds of bugs in one single issue, as one or the other might get lost in the discussion.

[ausi]

Your statements seem to tell me that everything is OK the way it is and it will stay that way.

How about this from https://github.com/contao/contao/issues/2398#issuecomment-703697185 :

However, I think it would make sense to be able to configure folders for either of these options through a core bundle configuration option as well.

@schnudd31do3 wouldn’t this fix your issue?

[schnudd31do3]

Could you please explain what do you mean with " core bundle configuration" and how it could be realized?

If this suggest would prevent from accessing unreadable folders like:
ls -la:
d????????? ? ? ? ? ? MyPrivateName
it would be the first necessary step to cooperate with mounted external Windows DFS systems.

And if your suggest means not to write the .public or .nosync or any .control file to archive folders, archive contents would no longer be modified. At the same time, file authorization errors like "Internal Server Error: Failed to touch ..." would no longer occur.

[fritzmg]

Could you please explain what do you mean with " core bundle configuration" and how it could be realized?

For example:

# config/config.yml
contao:
    dabfs:
        sync_exclude:
            - files/foobar
            - files/moobar

Or something like that.

If this suggest would prevent from accessing unreadable folders like:
ls -la:
d????????? ? ? ? ? ? MyPrivateName
it would be the first necessary step to cooperate with mounted external Windows DFS systems.

I am not exactly sure what you you mean here. Where do you want to prevent access to unreadable folders?

[schnudd31do3]

While I want to exclude unreadable folders from synchronizing, contao wants to write into unreadable folders which goeas fail.

So, this could only be prevented if the information for not-syncing is not stored in unreadable folders nor raedable folders where are no write permissens for contao. It only could be stored in configuration files of contao or in its database.

[fritzmg]

Why are you mounting "unreadable" folders into files/ in the first place? If they are unreadable, then you do not want them to show up in Contao (i.e. the file manager) at all, do you?

[schnudd31do3]

Our files are on a external Windows server which is a DFS. The DFS creates a unreadable folder which is used for syncing the underlaying servers. This unreadable folders are given by Windows and there is no way to make it invisible for the Linux share.

[schnudd31do3]

Our files are on a external Windows server which is a DFS. The DFS creates a unreadable folder which is used for syncing the underlaying servers. This unreadable folders are given by Windows and there is no way to make it invisible for the Linux share.
Duplicate of #

[schnudd31do3]

Wrongly closed because of duplicated post.

[m-vo]

The issues aside, I don't think it's a good idea to mount your "server sync metadata" (or whatever it is exactly) into the files accessible by your CMS. Why not just mount the data in the first place?

[fritzmg]

Our files are on a external Windows server which is a DFS. The DFS creates a unreadable folder which is used for syncing the underlaying servers. This unreadable folders are given by Windows and there is no way to make it invisible for the Linux share.

Yes, but couldn't you mount _just_ the readable folders into the files/ directory instead? As I said, you probably do not want the unreadable folders to be visible within Contao at all, do you?

You could probably even write a shell script that would mount any DFS folder - except specific ones.

[schnudd31do3]

No for many reasons. The suggests are only work aorunds. Why not simply fix the issues?

[m-vo]

No for many reasons. The suggests are only work aorunds.

No, they are simply not. Your idea of mounting metadata that doesn't have to do anything with your CMS is.

Why not simply fix the issues?

You didn't say anything to the idea that was mentioned, not did you help pointing to where the other issue occurs. There ain't such a thing as "simply fixing". If you've got an idea how to improve things, please make a PR. Input is very welcome but accusations aren't.

[schnudd31do3]

How do you think about this from above:

Could you please explain what do you mean with " core bundle configuration" and how it could be realized?

If this suggest would prevent from accessing unreadable folders like:
ls -la:
d????????? ? ? ? ? ? MyPrivateName
it would be the first necessary step to cooperate with mounted external Windows DFS systems.

And if your suggest means not to write the .public or .nosync or any .control file to archive folders, archive contents would no longer be modified. At the same time, file authorization errors like "Internal Server Error: Failed to touch ..." would no longer occur.

This verry concrete.

... not did you help pointing to where the other issue occurs. see my first post

[asaage]

i like @fritzmg approach
wich could probably be extended to hide resources completely from the BE:

# config/config.yml
contao:
    dabfs:
        sync_exclude:
            - files/foobar
            - files/moobar
        hidden:
            - files/hide_me
            - files/.*
        public:
            - files/pub*

with support for regex patterns of course - and maintainable via the contao-manager 😜

[fritzmg]

Why not simply fix the issues?

Because the issues you are experiencing are created by your environment, not by the web application.

[schnudd31do3]

@asage: Good idea. When I understand right, contao does not yet support this approach. Could you provide a test version?

[asaage]

Could you provide a test version?

nope 🙄

[bennyborn]

wich could probably be extended to hide resources completely from the BE

+1 for that idea. It would be nice to be able to keep the customer out of some folders simply by hiding them :)

[schnudd31do3]

If you would implement asaage's suggestion, writing .nosync is basically no longer necessary. Not writing the .nosync file would also be conform to the paradigm of never changing archive contents in any way.

Additionally it would be nice if you didn't have to edit the config.yml by hand, but via input fields in the administrator user interface.

Is it conceivable to implement the asaage's proposed solution in combination with an editing option via the administrator web interface together with not writing the .sync file in the next Contao version?

[schnudd31do3]

After a while I dealt with the problem again and wanted to see whether the asaage's solution from above was already working

I added

contao:
dabfs:
sync_exclude:
- files/var/www/html/contao/files/private/files/myfiles/folder_not_readable
hidden:
- files/var/www/html/contao/files/private/files/myfiles/folder_not_readable

to the parameters.yml.

When I click on "Dateisystem", the folder "folder_not_readable" is still visible. When I click on this folder to edit, I still get:

Forbidden
Was ist das Problem?
File or folder "files/private/files/ctlfiles/DfsrPrivate" is not mounted or cannot be found.

My Version is 4.10.4

[aschempp]

After a while I dealt with the problem again and wanted to see whether the asaage's solution from above was already working

That was a feature suggestion, nothing that has already been implemented.

Your statements seem to tell me that everything is OK the way it is and it will stay that way. Your proposed "solutions" do not match our understanding of IT security. Also not about the integrity of approved version statuses and also not about automated work processes. Nor do they solve the problem if contao wants to access folders with missing file information.

Yes, everything is OK in the sense that Contao works like this for years, and it works flawlessly for thousands of users. Your _understanding of IT security_ etc. seems to be different than what the rest of the web community and any hoster thinks.

If Contao has solved the two bugs I mentioned in my first post, we can use Contao in our company.

If this is a bug to you, you're very welcome to contribute a solution. Contao is Open Source, you're free to change the code and submit a suggestion. If this prevents you from using Contao, maybe Contao is not the correct solution for you.

We're willing to help you to use Contao, but you need to explain your server setup so we understand the use case. We can't just change something that works for everyone except you, without understanding what you're trying to do. Explain _why_ you have these readonly files in the CMS, and explain why you synchronize the folders at all please.

[asaage]

whether the asaage's solution from above was already working

That's just an idea.
It's not a solution, feature, or pull-request that's implemented yet (and i don't think it will be).
Wait for a label being applied to this issue!

[schnudd31do3]

@aschempp: "...understanding of IT security " is not the main thing here. The technical reasons are the significant ones. I have already explained the reasons here several times. If you could explain me a little where to adapt, I could try to and make a suggest.
@asaage: I know this and you did not promise to implement. But my second last post was not answered, so I had to beleve that you are thinking about to work on it.

[aschempp]

The technical reasons are the significant ones. I have already explained the reasons here several times.

Yes, unfortunately I don't understand them. You said there's a Windows server with readonly filesystem. I don't understand WHY this Windows filesystem is in your CMS, what files are on them that you need to publish etc.?

[schnudd31do3]

I will try once again: The files to deploy over a CMS are not on the local VM where the CMS is running. Those files are in a folder on a external DFS which is mounted on the VM. The DFS has that intrinsic not readable folder which I cannot hide nor leave alone.
It should be possible to exclude this folder from the synchronization and visibility, otherwise the errors described will occur. We cannot change our network structure only to get contao running.

[m-vo]

You could symlink just the files (not the unreadable folder) into the files folder of your Contao installation.

[schnudd31do3]

That would be too much of an expense. There are hundreds of files in tens of folders. Those folders is on the same level with the unreadable DFS folder. The contents changes dynamically each some days.

[bennyborn]

Not sure if I'm mistaken here but couldn't @schnudd31do3 `s case also be very similar to one when trying to deploy a Contao installation in a scalable environment with multiple hosts and a single read-only filesystem for static contents?

I'm thinking something like running multiple copies of the same Contao instance on AWS with all contents of /files hosted on an S3 storage?

This way I'd would be nice to have an option to exclude certain files and folders from synchronisation.

[asaage]

Interesting use-case @bennyborn.
Similar to mounting a complete next-cloud storage-folder under files where you wouldn't want everything to be syncronized, visible or writable.

[aschempp]

I will try once again: The files to deploy over a CMS are not on the local VM where the CMS is running. Those files are in a folder on a external DFS which is mounted on the VM. The DFS has that intrinsic not readable folder which I cannot hide nor leave alone.

I'm sorry, I still don't fully get it. Maybe you can try explaining your setup in german (assuming that's your native language)? Are you mounting the full CMS installation to another server or only some files into the /files directory?
Also, you didn't answer my question _why_ you synchronise the database at all?

[schnudd31do3]

@aschempp: I wrote you a PN under terminal42.ch in German :-)
@bennyborn: You are right. Your mentioned use-case is a good thing and it will also solve my problem. But it should be realized by keeping the excluding-information in a config-file or in the database because in my case there are no rights to write a ".nosync"-file zu the excluded folders.

[schnudd31do3]

In the meantime I tried to find out how to modify. This is not an advise or proven solution, only a test!
In my case the folders which must not be synchronized and most not be shown are named "DfsrPrivate".

Modificatios for invisibility:
contao/vendor/contao/core-bundle/src/Resources/contao/drivers/DC_Folder.php:
Below ...
foreach (Folder::scan($path) as $v)
{

I inserted:
if (preg_match('/DfsrPrivate/u', $v) === 1)
{
continue;
}
Now the "DfsrPrivate"-folders are invisible.

Without modifications for syncing I get this error: "Invalid resource files/private/files/xyz/DfsrPrivate"

Then I modified for syncing:
contao/vendor/contao/core-bundle/src/Resources/contao/library/Contao/Filter/SyncExclude.php
Below:
$strPath = $this->current()->getPathname();

I inserted:
if (preg_match('/DfsrPrivate/u', $strPath) === 1)
{
return false;
}

But then I receive an error: "the requested url was not found on this server". I could not find any hints in the logs from apache2 or contao why this fails.

[m-vo]

I guess you know that you should never modify files in the vendor folder.
This will break your ability to update forever.

You could symlink just the files (not the unreadable folder) into the files folder of your Contao installation.

That would be too much of an expense. There are hundreds of files in tens of folders.

I don't understand this problem. Just use globbing or write a simple bash script.

[asaage]

Just use globbing or write a simple bash script.

Agree for exclusion. But that would still not solve the problem of unwanted .nosync-files beeing created in folders that should not be synced.

[schnudd31do3]

@m-vo: I know that, therefore I said "...This is not an advise or proven solution, only a test!" But why does contao lose the requested url when I synchronize with excluded folders like I did?

A bash script which generates symlinks will not work because there are no rights to write on the shared archive folder and it also would cause high network traffic.

[m-vo]

A bash script which generates symlinks will not work because there are no rights to write on the shared archive folder and it also would cause high network traffic.

You're doing this on the webserver not on your share. And why would this generate high network traffic?

[schnudd31do3]

@m-vo: Once again: But why does contao lose the requested url when I synchronize with excluded folders like I did?

[m-vo]

I don't have the time to debug this for you nor do I think hacking this change into DC_Folder is beneficial.

[schnudd31do3]

Assage and I have said often enough how fundamental the solution should be. Why are there no contributions that further develop this direction?

Here it was said that contao was an open system. I'm happy to work with you, but I need people who don't tell me that they don't have time.

[leofeyer]

From my point of view it is a quick & dirty solution.

From our point of view it is the correct and intended solution ;). Folders are marked as not being synchronized through a .nosync file, public folders are marked through a .public file.

This is how I see it as well. If you have a CIFS share that is managed independently from Contao and therefore mounted read-only, you will have to add the .nosync and/or .public files on your host system and commit them in your VCS (if any).

[schnudd31do3]

@leofeyer: "...you will have to add the .nosync and/or .public files on your host system..." this is technical impossible when folders don't have write permissions and especially impossible when the folder has no read permossion. Please read all the posts to get the whole background.

[leofeyer]

Your CIFS share is certainly writeable on the host system, isn't it?