Tensorboard: Permission denied issue when writing to `/tmp/.tensorboard-info`

[tete1030]

• TensorBoard version (from pip package, also printed out when running tensorboard)
  1.13.1
• TensorFlow version if different from TensorBoard
  1.13.1
• OS Platform and version (e.g., Linux Ubuntu 16.04)
  Ubuntu 16.04
• Python version (e.g. 2.7, 3.5)
  2.7

Please describe the bug as clearly as possible, and if possible provide a minimal example (code, data, and/or command line) to reproduce the issue. Thanks!

The writing of tensorboard info files introduced in https://github.com/tensorflow/tensorboard/pull/1806 can cause permission problem under multi-user scenario. It directly create .tensorboard-info directory under /tmp as in https://github.com/tensorflow/tensorboard/blob/5fc3c8cea4b5f79c738345686a218f089b58ddba/tensorboard/manager.py#L237 . If the dir has already been created by an user, it will not be writable to other users.

[wchargin]

Hi @tete1030! Thanks for the clear report. This is a good point—I’d
considered the multi-user case and determined that it wouldn’t be a
problem for _reading_ files in this directory, but didn’t realize that
it would not be possible to _write_ new files in the directory.

I think that the following patch should suffice, at least on Unices:

diff --git a/tensorboard/manager.py b/tensorboard/manager.py
index a86c010b..92f7601f 100644
--- a/tensorboard/manager.py
+++ b/tensorboard/manager.py
@@ -235,6 +235,7 @@ def _get_info_dir():
   The directory will be created if it does not exist.
   """
   path = os.path.join(tempfile.gettempdir(), ".tensorboard-info")
+  old_umask = os.umask(0o000)
   try:
     os.makedirs(path)
   except OSError as e:
@@ -242,6 +243,8 @@ def _get_info_dir():
       pass
     else:
       raise
+  finally:
+    os.umask(old_umask)
   return path

I’ll have to test this on Windows. If you’re looking for a quick fix,
you should be able to patch your TensorBoard install as above. (Or just
chmod a+w /tmp/.tensorboard-info, which will work until the next time
that /tmp/ is cleared.)

[Harshini-Gadige]

@tete1030 Any update on this ?
@wchargin Please let me know if you want me to keep this issue open until Windows test ?

[tete1030]

Sorry that I didn't reply. The patch works great and I have not encountered this problem again.

[wchargin]

Great—glad to hear that the patch is working, @tete1030 (and sorry for
the inconvenience).

@hgadig: Yes, please keep this open.

[lebrice]

Hey there, just stumbled upon this, I just thought I'd mention that this patch doesn't fix the issue from the non-sudo user perspective. I added the patch first, but It did not fix the problem, as I do not have write access on that directory anyway. I'm gonna try and change the ".tensorboard-info" name to something unique, and hope this works. I'm thinking it might be nice to be able to customize this location, on a per-user basis ?

[wchargin]

@lebrice, could you clarify what you mean by the “non-sudo user
perspective”? I can see from @tete1030’s screenshot that they’re not
running as root or with sudo. Does your user account not have write
access to $TMPDIR?

(To be clear, the patch needs to be applied _before_ the info directory
is first created. If you’ve created a write-restricted info directory as
root by running TensorBoard without this patch, then yes, you’ll need to
remove it or change its mode.)

[noisychannel]

My workaround while this issue is resolved involves setting the TMPDIR environment variable. Since tensorboard uses tempfile which will respect user set environment variables (TMPDIR, TMP, etc.), this is possible. Make sure that the directory which TMPDIR points to exists!

You can change and test the new temp directory by running the following (which sets it to /tmp/$USER instead of /tmp).

export TMPDIR=/tmp/$USER; mkdir -p $TMPDIR; python -c "import tempfile; print(tempfile.gettempdir())"

A safe tensorboard invocation is:

export TMPDIR=/tmp/$USER; mkdir -p $TMPDIR; tensorboard --logdir $LOGDIR

[noisychannel]

Also, for @wchargin, the issue being referenced as the multi-user scenario is:

1. User A starts tensorboard and hence /tmp/.tensorboard-info is owned by that user.
2. When user B invokes tensorboard, it will crash since it cannot create files in /tmp/.tensorboard-info which is owned by user A. This ownership cannot be changed without sudo access.

[wchargin]

Also, for @wchargin, the issue being referenced as the multi-user
scenario is:

Right, I understand this; thanks. (That’s what this issue is about.) And
this should be fixed by the patch above, though changing TMPDIR is
also certainly a valid workaround.

[dby2017]

@wchargin I have encountered this problem and used the method you provided. I still can't solve it.

[tete1030]

@wchargin I have encountered this problem and used the method you provided. I still can't solve it.

Make sure you have deleted the old /tmp/.tensorboard-info, or use chmod a+w /tmp/.tensorboard-info. They both require root privilege.

Be aware if you have only patched tensorboard installed in one environment (e.g. a conda env, or locally installed package), other user with unpatched env could also cause this problem

[minygd]

@wchargin I have encountered this problem and used the method you provided. I still can't solve it.

Make sure you have deleted the old /tmp/.tensorboard-info, or use chmod a+w /tmp/.tensorboard-info. They both require root privilege.

Be aware if you have only patched tensorboard installed in one environment (e.g. a conda env, or locally installed package), other user with unpatched env could also cause this problem

So does it means that in one Linux Server, only one user can use tensorboard freely? And if others want to use it, he/she must get root privilege to fix the problem?

[wchargin]

@minygd: This issue has been fixed, and the fix will be in the next
TensorBoard release. You can get that release now by installing the
latest version of tb-nightly, or you can wait for TensorBoard 1.14 to
be released (in the next few weeks, probably).

[reactivetype]

@wchargin I am using ubuntu with tb-nightly 1.14.0a20190506 and I still have the issue.

File "envs/tf-nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 269, in write_info_file
    with open(_get_info_file_path(), "w") as outfile:
PermissionError: [Errno 13] Permission denied: '/tmp/.tensorboard-info/pid-10141.info'

[wchargin]

@reactivetype: Could you please run

pip freeze 2>&1 | grep -e tensor -e tf- -e tb-; printf '%s\n' ---; python -c 'print(__import__("inspect").getsource(__import__("tensorboard.manager").manager._get_info_dir))'; printf '%s\n' ---; stat /tmp/.tensorboard-info/

and post the full output here?

[reactivetype]

@wchargin thanks for replying. here is the output:

$ pip freeze 2>&1 | grep -e tensor -e tf- -e tb-; printf '%s\n' ---; python -c 'print(__import__("inspect").getsource(__import__("t ensorboard.manager").manager._get_info_dir))'; printf '%s\n' ---; stat /tmp/.tensorboard-info/
tb-nightly==1.14.0a20190506
tf-estimator-nightly==1.14.0.dev2019042901
tf-nightly-gpu==1.14.1.dev20190506
---
def _get_info_dir():
  """Get path to directory in which to store info files.

  The directory returned by this function is "owned" by this module. If
  the contents of the directory are modified other than via the public
  functions of this module, subsequent behavior is undefined.

  The directory will be created if it does not exist.
  """
  path = os.path.join(tempfile.gettempdir(), ".tensorboard-info")
  try:
    os.makedirs(path)
  except OSError as e:
    if e.errno == errno.EEXIST and os.path.isdir(path):
      pass
    else:
      raise
  else:
    os.chmod(path, 0o777)
  return path

---
  File: '/tmp/.tensorboard-info/'
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: fc00h/64512d    Inode: 9961546     Links: 2
Access: (0775/drwxrwxr-x)  Uid: ( 1001/a00447759)   Gid: ( 1001/a00447759)
Access: 2019-04-02 21:38:07.367113664 -0400
Modify: 2019-04-22 12:22:51.280130190 -0400
Change: 2019-04-22 12:22:51.280130190 -0400
 Birth: -

[wchargin]

@reactivetype: Thanks. It looks like your .tensorboard-info directory
was created with an older version of TensorBoard, before the fix went
in. You’ll need to manually change its permissions just this once:

sudo chmod 777 /tmp/.tensorboard-info

[reactivetype]

@wchargin Makes sense. Thank you very much!

[zhixuanli]

Just try

sudo chmod 777 /tmp/.tensorboard-info/

[alwynmathew]

@reactivetype: Thanks. It looks like your .tensorboard-info directory
was created with an older version of TensorBoard, before the fix went
in. You’ll need to manually change its permissions just this once:

sudo chmod 777 /tmp/.tensorboard-info

hey @wchargin I have installed tensorboard 1.14 but I face the same issues. Seems like I have a .tensorboard-info directory created with an older version of TensorBoard. As I'm not a sudoer, I dont have permission to delete or change the permission of /tmp/.tensorboard-info/. Is there a way to fix it without going to our admin requesting to delete or change the permission of /tmp/.tensorboard-info/?

[wchargin]

@alwynmathew: Who owns the /tmp/.tensorboard-info directory? (Find out
with stat --format=%U /tmp/.tensorboard-info.) That user—probably
whoever created the directory—can chmod it or remove it.

[SimonSelg]

@alwynmathew: I have encountered the same problem while working on a shared machine. Since the admin is on christmas break, I had to find another solution.

Turns out, tensorboard uses tempfile.tempdir to get the path to the temporary directory (usually /tmp/). In this directory the .tensorboard folder is created.

It is possible to specify the temp directory returned by tempfile.tempdir by setting the environment variable TMPDIR.

Now we can simply force tensorboard to use another directory then /tmp/.tensorboard/, like /tmp/.selgs/.tensorboard:

mkdir /tmp/.selgs
env TMPDIR=/tmp/.selgs tensorboard 

[wchargin]

Yep, using TMPDIR in this format is specifically encouraged by
POSIX. That’s a good workaround; thanks for sharing!

(You probably want to use either /tmp/selgs or /tmp/.selgs in both
lines, and you can also just use TMPDIR=/tmp/selgs tensorboard to set
an environment variable for one command; no need to use env, which
spawns an extra process.)

[alwynmathew]

@alwynmathew: I have encountered the same problem while working on a shared machine. Since the admin is on christmas break, I had to find another solution.

Turns out, tensorboard uses tempfile.tempdir to get the path to the temporary directory (usually /tmp/). In this directory the .tensorboard folder is created.

It is possible to specify the temp directory returned by tempfile.tempdir by setting the environment variable TMPDIR.

Now we can simply force tensorboard to use another directory then /tmp/.tensorboard/, like /tmp/.selgs/.tensorboard:

mkdir /tmp/selgs
env TMPDIR=/tmp/.selgs tensorboard 

Sorry @SimonSelg , I was in Christmas break too. @wchargin latest solution seems elegant with just one command. I had a tough time fixing my issues, I tracked down the owner of the dir with stat --format=%U /tmp/.tensorboard-info then pinged him to change the permission. Hope you found the solution.