Packit-service: Getting archive path from user action does not work in service

[lachmanfrantisek]

Packit-service does not find the archive path in the output of the create_archive user action, unlike the packit CLI.

• action is run in the sandbox, checking is done in the worker => can this be a problem when we use absolute paths? (ping @TomasTomecek )
• example PR: https://github.com/psss/tmt/pull/249 (@psss )
  
  
  
  • logs: https://prod.packit.dev/srpm-build/2901/logs
  
  
• works well localy:

$ /usr/bin/packit --debug srpm
Using user-defined script for ActionName.create_archive: [['make', 'tarball']]
08:59:08.563 utils.py          DEBUG  cmd = 'make tarball'
08:59:08.570 utils.py          DEBUG  rm -rf /home/flachman/Projects/psss/tmt/tmp build dist .cache .pytest_cache
08:59:08.581 utils.py          DEBUG  rm -rf docs/_build docs/stories docs/spec
08:59:08.583 utils.py          DEBUG  find . -type f -name "*.py[co]" -delete
08:59:08.590 utils.py          DEBUG  find . -type f -name "*,cover" -delete
08:59:08.597 utils.py          DEBUG  find . -type d -name "__pycache__" -delete
08:59:08.605 utils.py          DEBUG  rm -f .coverage tags
08:59:08.607 utils.py          DEBUG  rm -f examples/convert/main.fmf
08:59:08.609 utils.py          DEBUG  mkdir -p /home/flachman/Projects/psss/tmt/tmp/.fmf
08:59:08.619 utils.py          DEBUG  mkdir -p /home/flachman/Projects/psss/tmt/tmp/SOURCES
08:59:08.620 utils.py          DEBUG  mkdir -p /home/flachman/Projects/psss/tmt/tmp/tmt-0.16
08:59:08.622 utils.py          DEBUG  cp -a LICENSE README.rst Makefile tmt.spec setup.py examples tmt bin tests /home/flachman/Projects/psss/tmt/tmp/tmt-0.16
08:59:08.642 utils.py          DEBUG  rm /home/flachman/Projects/psss/tmt/tmp/tmt-0.16/tmt/steps/provision/{base,vagrant}.py
08:59:08.647 utils.py          DEBUG  cp docs/header.txt /home/flachman/Projects/psss/tmt/tmp/man.rst
08:59:08.648 utils.py          DEBUG  tail -n+7 README.rst >> /home/flachman/Projects/psss/tmt/tmp/man.rst
08:59:08.649 utils.py          DEBUG  rst2man /home/flachman/Projects/psss/tmt/tmp/man.rst > /home/flachman/Projects/psss/tmt/tmp/tmt-0.16/tmt.1
08:59:09.275 utils.py          DEBUG  cd /home/flachman/Projects/psss/tmt/tmp && tar cfz SOURCES/tmt-0.16.tar.gz tmt-0.16
08:59:09.296 utils.py          DEBUG  /home/flachman/Projects/psss/tmt/tmp/SOURCES/tmt-0.16.tar.gz
08:59:09.298 base_git.py       DEBUG  Action command output: ['rm -rf /home/flachman/Projects/psss/tmt/tmp build dist .cache .pytest_cache\nrm -rf docs/_build docs/stories docs/spec\nfind . -type f -name "*.py[co]" -delete\nfind . -type f -name "*,cover" -delete\nfind . -type d -name "__pycache__" -delete\nrm -f .coverage tags\nrm -f examples/convert/main.fmf\nmkdir -p /home/flachman/Projects/psss/tmt/tmp/.fmf\nmkdir -p /home/flachman/Projects/psss/tmt/tmp/SOURCES\nmkdir -p /home/flachman/Projects/psss/tmt/tmp/tmt-0.16\ncp -a LICENSE README.rst Makefile tmt.spec setup.py examples tmt bin tests /home/flachman/Projects/psss/tmt/tmp/tmt-0.16\nrm /home/flachman/Projects/psss/tmt/tmp/tmt-0.16/tmt/steps/provision/{base,vagrant}.py\ncp docs/header.txt /home/flachman/Projects/psss/tmt/tmp/man.rst\ntail -n+7 README.rst >> /home/flachman/Projects/psss/tmt/tmp/man.rst\nrst2man /home/flachman/Projects/psss/tmt/tmp/man.rst > /home/flachman/Projects/psss/tmt/tmp/tmt-0.16/tmt.1\ncd /home/flachman/Projects/psss/tmt/tmp && tar cfz SOURCES/tmt-0.16.tar.gz tmt-0.16\n/home/flachman/Projects/psss/tmt/tmp/SOURCES/tmt-0.16.tar.gz\n']
Created archive: /home/flachman/Projects/psss/tmt/tmp/SOURCES/tmt-0.16.tar.gz
Linking to the specfile directory: /home/flachman/Projects/psss/tmt/tmt-0.16.tar.gz
08:59:09.300 base_git.py       DEBUG  Running ActionName.fix_spec.
08:59:09.300 base_git.py       DEBUG  Running default implementation for ActionName.fix_spec.
06:59:09.433 upstream.py       DEBUG  new %setup line:
%setup -q -n tmt-0.16
06:59:09.434 utils.py          DEBUG  cmd = 'git describe --tags --long --match *'
06:59:09.442 utils.py          DEBUG  0.16-3-ge2567ed
06:59:09.455 upstream.py       DEBUG  Setting Release in spec to '1.20200521065909454919.3.ge2567ed'
06:59:09.978 upstream.py       DEBUG  SRPM build command: "rpmbuild" "-bs" "--define" "_sourcedir /home/flachman/Projects/psss/tmt" "--define" "_srcdir /home/flachman/Projects/psss/tmt" "--define" "_specdir /home/flachman/Projects/psss/tmt" "--define" "_srcrpmdir /home/flachman/Projects/psss/tmt" "--define" "_topdir /home/flachman/Projects/psss/tmt" "--define" "_builddir /home/flachman/Projects/psss/tmt" "--define" "_rpmdir /home/flachman/Projects/psss/tmt" "--define" "_buildrootdir /home/flachman/Projects/psss/tmt" "tmt.spec"
06:59:09.981 upstream.py       DEBUG  present srpms: set()
06:59:09.982 utils.py          DEBUG  cmd = 'rpmbuild -bs --define _sourcedir /home/flachman/Projects/psss/tmt --define _srcdir /home/flachman/Projects/psss/tmt --define _specdir /home/flachman/Projects/psss/tmt --define _srcrpmdir /home/flachman/Projects/psss/tmt --define _topdir /home/flachman/Projects/psss/tmt --define _builddir /home/flachman/Projects/psss/tmt --define _rpmdir /home/flachman/Projects/psss/tmt --define _buildrootdir /home/flachman/Projects/psss/tmt tmt.spec'
06:59:10.111 utils.py          DEBUG  setting SOURCE_DATE_EPOCH=1590019200
06:59:10.118 utils.py          DEBUG  Wrote: /home/flachman/Projects/psss/tmt/tmt-0.16-1.20200521065909454919.3.ge2567ed.fc31.src.rpm
06:59:10.125 upstream.py       DEBUG  setting SOURCE_DATE_EPOCH=1590019200
Wrote: /home/flachman/Projects/psss/tmt/tmt-0.16-1.20200521065909454919.3.ge2567ed.fc31.src.rpm
SRPM is /home/flachman/Projects/psss/tmt/tmt-0.16-1.20200521065909454919.3.ge2567ed.fc31.src.rpm
SRPM: /home/flachman/Projects/psss/tmt/tmt-0.16-1.20200521065909454919.3.ge2567ed.fc31.src.rpm

[sentry-io[bot]]

Sentry issue: RED-HAT-0P-24D

[mfocko]

For future reference

Directly related comments

• https://github.com/psss/tmt/pull/249#issuecomment-666599696
• https://github.com/psss/tmt/pull/249#issuecomment-668545068

Related PR in packit

packit/packit#923

Summary

• scripts MUST return relative path, since it's always prefixed with local_project.working_dir when parsing
• Added extended logging and enforced converting paths
  before: symlinks archive in specfile_dir to _local_project.working_dir / archive_name
  now: parsing path to archive returns absolute path and then before linking is converted to relative path
• from logs:
  STDOUT: ./tmp/SOURCES/tmt-0.20.tar.gz Created archive: relative path: /sandcastle/tmp/SOURCES/tmt-0.20.tar.gz absolute path: /sandcastle/tmp/SOURCES/tmt-0.20.tar.gz Linking to the specfile directory: /sandcastle/tmt-0.20.tar.gz -> tmp/SOURCES/tmt-0.20.tar.gz (absolute path to archive: /sandcastle/tmp/SOURCES/tmt-0.20.tar.gz)
/sandcastle/tmt-0.20.tar.gz -> /sandcastle/tmp/SOURCES/tmt-0.20.tar.gz (symlink before, failing)
  /sandcastle/tmt-0.20.tar.gz -> tmp/SOURCES/tmt-0.20.tar.gz (symlink now, passing)

---

@lachmanfrantisek can we close?

[lachmanfrantisek]

scripts MUST return relative path, since it's always prefixed with local_project.working_dir when parsing

Can we read and convert somehow the absolute paths as well? (And require only to be somewhere in the repo dir?)

@lachmanfrantisek can we close?

Above + Can you please check the docs that we have it clearly described?

[TomasTomecek]

+1 on making sure we have this covered in docs

this should also be doable to convert to a relative path since we know that if it's absolute, it will always start with /sandcastle (the only writable place in the sandbox)

[mfocko]

create-archive - is expected to return the created archive name. If there are more steps, then one of them has to return the archive name.

^ from docs; sould be changed to explicitly require relative path for now?

Regarding absolute path from scripts I could create a follow-up issue. (Should be just quick branching between relative and absolute; more like optional prefixing with local_project.working_dir).

[TomasTomecek]

_create-archive_ - is expected to return the created archive name. If there are more steps, then one of them has to return the archive name.

^ from docs; sould be changed to explicitly require relative path for now?

Yes, please.

Regarding absolute path from scripts I could create a follow-up issue. (Should be just quick branching between relative and absolute; more like optional prefixing with local_project.working_dir).

:+1:

[jkonecny12]

Thanks a lot for solving this! I will try it on my project instead of copying the archive ;)