Azure-cli: [CloudShell] Error on every command run on Azure Cloud Shell: OSError: [Errno 30] Read-only file system

[PowellTravis]

This is autogenerated. Please review and update as needed.

Describe the bug

When I go to run any command on the system it errors out and prevents me from using the cloud shell interface.

Command Name
Any command at all

Errors:

The command failed with an unexpected error. Here is the traceback:

[Errno 30] Read-only file system: '/home/**Redacted Username**/.azure/commands/2020-06-05.22-24-38.unknown_command.211.log'
Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 215, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 511, in execute
    load_cmd_tbl_func=self.commands_loader.load_command_table, args=args)
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 164, in raise_event
    func(self, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/azlogging.py", line 86, in init_command_file_logging
    self._init_command_logfile_handlers(cmd_logger, args)  # pylint: disable=protected-access
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/azlogging.py", line 108, in _init_command_logfile_handlers
    logfile_handler = logging.FileHandler(log_file_path)
  File "/opt/az/lib/python3.6/logging/__init__.py", line 1032, in __init__
    StreamHandler.__init__(self, self._open())
  File "/opt/az/lib/python3.6/logging/__init__.py", line 1061, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
OSError: [Errno 30] Read-only file system: '/home/**Redacted Username**/.azure/commands/2020-06-05.22-24-38.unknown_command.211.log'

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• I don't know how to reproduce the errors. I just get the error every time I put in a command.

Expected Behavior

I expected all the commands to not error out on run.

Environment Summary

Cloud Shell

Additional Context

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @maertendMSFT.

[yonzhan]

Cloud Shell

[jiasli]

This issue has been mentioned multiple times.

Looks like you somehow lost permission to /home/Redacted Username/.azure/commands/2020-06-05.22-24-38.unknown_command.211.log.

13848 is worse - even permission to /home/username/.azure/az.sess is lost:

OSError: [Errno 30] Read-only file system: '/home/username/.azure/az.sess'

Could you check the permission of this folder?

$ stat ~/.azure/commands
  File: '/home/leo/.azure/commands'
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 700h/1792d      Inode: 131079      Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 9527/     leo)   Gid: ( 9527/     leo)
Access: 2020-06-09 08:11:16.161186717 +0000
Modify: 2020-06-09 08:10:56.481010326 +0000
Change: 2020-06-09 08:10:56.481010326 +0000
 Birth: -

$ ls ~/.azure/commands -l
total 92
-rw-r--r-- 1 leo leo 472 Jun  3 02:38 2020-06-03.02-38-06.ad_sp_credential_reset.694.log
-rw-r--r-- 1 leo leo 480 Jun  3 02:38 2020-06-03.02-38-13.ad_sp_credential_reset.739.log
-rw-r--r-- 1 leo leo 210 Jun  5 07:41 2020-06-05.07-41-35.login.103.log
-rw-r--r-- 1 leo leo 223 Jun  5 07:41 2020-06-05.07-41-38.account_set.116.log
-rw-r--r-- 1 leo leo 536 Jun  5 07:41 2020-06-05.07-41-43.rest.289.log

Also, could you try creating a file with

$ echo "Hello World" > ~/.azure/commands/test.txt

$ cat ~/.azure/commands/test.txt
Hello World

[PowellTravis]

I did the commands these were my outputs:

$ stat ~/.azure/commands
  File: '/home/**Redacted Username**/.azure/commands'
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 700h/1792d      Inode: 131079      Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 9527/**Redacted Username**)   Gid: ( 9527/**Redacted Username**)
Access: 2020-06-09 14:38:02.020203476 +0000
Modify: 2020-06-09 14:38:01.504197941 +0000
Change: 2020-06-09 14:38:01.504197941 +0000
 Birth: -

$ ls ~/.azure/commands -l
total 32
-rw-r--r-- 1 **Redacted** **Redacted** 210 Jun  5 22:30 2020-06-05.22-30-36.login.115.log
-rw-r--r-- 1 **Redacted** **Redacted** 223 Jun  5 22:30 2020-06-05.22-30-39.account_set.126.log
-rw-r--r-- 1 **Redacted** **Redacted** 194 Jun  5 23:23 2020-06-05.23-23-58.unknown_command.183.log
-rw-r--r-- 1 **Redacted** **Redacted** 404 Jun  5 23:24 2020-06-05.23-24-12.aks_install-cli.190.log
-rw-r--r-- 1 **Redacted** **Redacted** 210 Jun  9 14:38 2020-06-09.14-37-59.login.103.log
-rw-r--r-- 1 **Redacted** **Redacted** 223 Jun  9 14:38 2020-06-09.14-38-01.account_set.115.log
-rw-r--r-- 1 **Redacted** **Redacted** 194 Jun  9 14:38 2020-06-09.14-38-50.unknown_command.140.log
-rw-r--r-- 1 **Redacted** **Redacted** 404 Jun  9 14:39 2020-06-09.14-39-19.aks_install-cli.145.log

Command with test.txt

$echo "Hello World" > ~/.azure/commands/test.txt
([Errno 13] Permission denied: '~/.azure/commands/test.txt')
$ cat ~/.azure/commands/test.txt
~/.azure/commands/test.txt does not exist

I had also tried to install az aks install-cli and I got this out

Downloading client to "/usr/local/bin/kubectl" from "https://storage.googleapis.com/kubernetes-release/release/v1.18.3/bin/linux/amd64/kubectl"
Connection error while attempting to download client ([Errno 13] Permission denied: '/usr/local/bin/kubectl')

[smcgrawDotNet]

Good day,

Different command; however, same error as above:

{OSError: [Errno 30] Read-only file system: '/home/samuel/.azure/az.sess'}

samuel@Azure:~$ echo "hello world" > ~/.azure/commands/test.txt
bash: /home/samuel/.azure/commands/test.txt: Read-only file system

NOTE: Access was working until I deleted two 'Resource groups' via the gui.
I am working from the 'cloud shell' in 'Edge'.

[jiasli]

Thanks for the information! I was wonder if you still have permission to the home folder. Could you try to write something to upper locations?

echo "hello world" > ~/.azure/test.txt
echo "hello world" > ~/test.txt

@PowellTravis, is **Redacted** yourself as shown in the terminal's prompt?

leo@Azure:~$

[PowellTravis]

Sorry for the late reply. Yes Redacted is myself.

[jiasli]

Hi @PowellTravis, how about the operations I mentioned above? Could you try to write to the home folder as well?

echo "hello world" > ~/.azure/test.txt
echo "hello world" > ~/test.txt

[Benoitlotfallah]

Hi @jiasli i have the same problem, and couldn't write on any of them

benoit@Azure:~$ echo "hello world" > ~/.azure/test.txt
bash: /home/benoit/.azure/test.txt: Read-only file system
benoit@Azure:~$ echo "hello world" > ~/test.txt
bash: /home/benoit/test.txt: Read-only file system

[jiasli]

Thanks @Benoitlotfallah for the information. We can isolate the issue to Cloud Shell itself. I will work with Cloud Shell team to see how we can address it.

[edyoung]

Unfortunately, it looks as though the file which stores your home directory _may_ have become corrupted. However I don't know exactly how so we'll need to investigate a bit.

Can you (any user experiencing this problem) confirm:

• does this issue persist across multiple Cloud Shell sessions?
• if you are comfortable doing so, please share the value of "echo $ACC_PUID" - we can use this to identify the sessions you created and there might be information in the logs which will help

To work around the issue, you can delete and recreate your home directory, like this:

1) this will result in deleting any files which are under ~ apart from files which are in ~/clouddrive. Copy any files you need to keep under the ~/clouddrive directory

2) In the Cloud Shell toolbar, Click Upload/Download > Manage File Share

3) You'll see in the pane above Cloud Shell a view of your Cloud Drive file share, including a directory called .cloudconsole. Open that and you'll see a file called something like acc_username.img

4) In the Cloud Shell toolbar, click 'Restart Cloud Shell' and close the Cloud Shell pane before it starts a new shell. (This is so we unmount your home directory)

5) Delete the file acc_username.img.

6) Start Cloud Shell again. We will recreate a new home directory for you. The directory should be writable including the .azure directory. You can copy saved files from ~/clouddrive back into the home directory.

[PowellTravis]

This issue has persisted across all of my cloud shells, and kept up even when I delete the cloud shell storage and create a new one.

[edyoung]

@PowellTravis that's quite strange. Nothing is actually persisted outside the storage so I'm confused how that works.

When you say 'delete the cloud shell storage', do you mean actually delete the storage account or file share via the portal, or run 'clouddrive unmount'? If you run the latter and then reselect the same file share when you reconnect to cloud shell, you'd still be using the same files. Might be worth checking the date on that .img file I mention above.

[PowellTravis]

I have deleted the storage account where the cloud shell data is stored.

[edyoung]

Do you have write access to files in the ~/clouddrive directory?
When you recreated the cloud drive storage after deleting it, did you see this 'access denied' issue immediately?
If you try the steps I list above, does it make a difference? (If deleting the storage account didn't, it's unlikely)

[PowellTravis]

I do have access to the ~/clouddrive directory. That is the only thing I have the permission to view on the system still. I do get the access denied even after deleting and recreating the storage account.

[Herve-Stephane]

Hello,

Get the same error trying first to create AKS and now az login doesn't work anymore.

Stef

[smcgrawDotNet]

Good day,

Confirm (or fix) your login first.
Make sure you have:

• Close and open a new ACS tab;
• Some-times ACS works better in Chrome than IE;
• In both cases clear the browser cache (logout first - after
  logback-in);

Sam

On Tue, Jul 28, 2020 at 7:29 AM Herve-Stephane [e-mail]
wrote:

Hello,

Get the same error trying first to create AKS and now az login doesn't
work anymore.

Stef

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-cli/issues/13855#issuecomment-665074580,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHNMBYFHDDXFQF2W3UIJR2DR53OFVANCNFSM4NVIQNGA
.

[edyoung]

I don't believe this would be resolved by reopening Cloud Shell or using an alternative browser. Please see my comment earlier in this thread for troubleshooting steps.

[Herve-Stephane]

Hello, Thanks.

Sam it's the first thing i've done. Restarting everything.
Ed i can't do it by command line but with the portal it works. I will try to recreate the directory as you explain.

Thanks again

[Herve-Stephane]

Hi,

This morning everything works fine. To be continued...

[Herve-Stephane]

Sorry in fact when i tried to write a file it works but nothing more.

[maertendMSFT]

@Herve-Stephane so you are only able to create a file, but you are still getting issues when running commands?

[patricekrakow]

Yes, I can reproduce the problem and a fix/workaround:
1/ Delete the group "cloud-shell-storage-westeurope" in which the "Storage account" for the Cloud Shell has been created.
2/ Wait for the deletion.
3/ Go back to Cloud Shell and click "Create storage".
4/ See the issue by typing (for instance):

$ echo "hello world" > ~/test.txt
bash: /home/benoit/test.txt: Read-only file system

5/ Restart Cloud Shell with the "power" button/icon.
6/ Check that the issue is fixed by typing (for instance):

$ echo "hello world" > ~/test.txt

Hope this help.

[smcgrawDotNet]

Good day,

Same problem -- I had to close and reopen any tools connected to Azure -- I
was leaving my system locked at the end of the day, however, I would leave
tools open.

On Thu, Nov 5, 2020 at 6:12 AM Patrice Krakow [e-mail]
wrote:

I had the same issue "Read-only file system", and it (might) has been
fixed by restarting the cloud shell using the "power" button.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-cli/issues/13855#issuecomment-722402557,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHNMBYB4QR5PIP4VEHVUNTTSOKXDBANCNFSM4NVIQNGA
.

[maertendMSFT]

@patricekrakow does this only occur the first time you are connecting to the new file share? My guess is that you are deleting the old file share, creating the new file share, then cloud shell is giving you the session before it has finished making the updates to the storage account settings for the new file share.

[dempsey93]

Yes, I can reproduce the problem and a fix/workaround:
1/ Delete the group "cloud-shell-storage-westeurope" in which the "Storage account" for the Cloud Shell has been created.
2/ Wait for the deletion.
3/ Go back to Cloud Shell and click "Create storage".
4/ See the issue by typing (for instance):

$ echo "hello world" > ~/test.txt
bash: /home/benoit/test.txt: Read-only file system

5/ Restart Cloud Shell with the "power" button/icon.
6/ Check that the issue is fixed by typing (for instance):

$ echo "hello world" > ~/test.txt

Hope this help.

[dempsey93]

It works perfectly, thanks a lot :)

[balajiraman]

Yes, I can reproduce the problem and a fix/workaround:
1/ Delete the group "cloud-shell-storage-westeurope" in which the "Storage account" for the Cloud Shell has been created.
2/ Wait for the deletion.
3/ Go back to Cloud Shell and click "Create storage".
4/ See the issue by typing (for instance):

$ echo "hello world" > ~/test.txt
bash: /home/benoit/test.txt: Read-only file system

5/ Restart Cloud Shell with the "power" button/icon.
6/ Check that the issue is fixed by typing (for instance):

$ echo "hello world" > ~/test.txt

Hope this help.

I have tried this, does not work.

When I deleted my storage account and recreated the shell.. its mounting a ready only drive. What I find using df command is that it does not mount he cloud shell file share. When I restart cloudshell, it hangs with black screen for ever, no error messages to be seen. I have done multiple times, with autogenerated CS namespaces to creating my own with no avail. is this specific user related issue with a hung process somewhere in the stack?