Awx: Git project that contains submodules with relative paths or authentication is not working

[tamirshaul]

ISSUE TYPE

• Bug Report
  When using a git repo with submodules (both relative submodule path and full path) AWX fails when running a job template that uses the git project.

SUMMARY

AWX tries to clone the local repo to /tmp with this command:

git clone --branch=awx_local/<random_uid> --depth=1 --recursive --single-branch -v file:///var/lib/awx/projects/_7__My_Repo /tmp/awx_<random_dir>/project

When using full paths - the problem is that the git credential is not passed to clone the submodules.

When using relative paths (for example ../sub_repo or ../sub_repo.git) - AWX tries to clone the submodule from /var/lib/awx/projects/sub_repo which is also not working.

The problem is not with syncing the Project itself but with running a job template that uses this project.

ENVIRONMENT

• AWX version: 7.0.0
• AWX install method: openshift
• Ansible version: 2.7.5
• Operating System: Red Hat Enterprise Linux 7.6
• Web Browser: Tried on both Chrome and Firefox

STEPS TO REPRODUCE

• Create a project in awx that uses a git repo that contains submodules
• Try to run a job that uses this project
  

  EXPECTED RESULTS

Job should start normally

ACTUAL RESULTS

Job fails before even starting (Error details on the DETAILS section)
Sorry for no print screen, we are running on an On prem environment.

ADDITIONAL INFORMATION

On prem environment

[wenottingham]

Did this work in a prior release?

[AlanCoding]

The relative paths issue could be https://github.com/gitpython-developers/GitPython/issues/730, I can see how we will probably have to do hacks around this.

[tamirshaul]

Works in 6.1.0, the issue appears only in 7.0.0

[AlanCoding]

reproduced relative submodules with https://github.com/alancoding/has-relative-submodule

[AlanCoding]

I put up a proposed patch. The issue I referenced in my prior comment, https://github.com/ansible/awx/issues/4703#issuecomment-530046456, would not resolve this problem if it were fixed.

[alancivita]

Following this one.
Same problem here, works in 6.0.0 and not working in 7.0.0 , my project contains submodules as well.

[mwiora]

Same problem here - I've seen there was a merge on Friday from issue #4735 - so I've tested the new current devel state (commit d2a5af44) and the git submodules authentication issue seems to persist.

[AlanCoding]

Thanks for the info. Could you give an updated traceback or project update stdout with this recent version that you just tested it with? That would help me narrow down the details of reproducing it.

[mwiora]

sure :)

Traceback (most recent call last): 

File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 1174, in run self.pre_run_hook(self.instance, private_data_dir) 

File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 1716, in pre_run_hook recursive=True # include submodules 

File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/git/repo/base.py", line 988, in clone_from return cls._clone(git, url, to_path, GitCmdObjectDB, progress, **kwargs) 

File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/git/repo/base.py", line 939, in _clone finalize_process(proc, stderr=stderr) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/git/util.py", line 333, in finalize_process proc.wait(**kwargs) 

File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/git/cmd.py", line 415, in wait raise GitCommandError(self.args, status, errstr) git.exc.GitCommandError: Cmd('git') failed due to: exit code(1) cmdline: git clone --branch=awx_internal/bdf87c59-8d52-473a-ae5d-47303ba466bc --depth=1 --recursive --single-branch -v file:///var/lib/awx/projects/_8__bt_dev /tmp/awx_9_yvvgpb89/project stderr: 'Cloning into '/tmp/awx_9_yvvgpb89/project'... Submodule 'playbooks' ([email protected]:PROJECT/somerepo.git) registered for path 'playbooks' Cloning into '/tmp/awx_9_yvvgpb89/project/playbooks'... 

Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. fatal: clone of '[email protected]:PROJECT/somerepo.git' into submodule path '/tmp/awx_9_yvvgpb89/project/playbooks' failed Failed to clone 'playbooks'. Retry scheduled Cloning into '/tmp/awx_9_yvvgpb89/project/playbooks'... Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. fatal: clone of '[email protected]:PROJECT/somerepo.git' into submodule path '/tmp/awx_9_yvvgpb89/project/playbooks' failed Failed to clone 'playbooks' a second time, aborting '

tried to format it a little bit....
note: a clone with the same key works perfectly for the affected repository including all submodules.

[mwiora]

step backwards - it seems my described issue is introduced by using tower-cli to import a backup.json of credentials/jobs. A manually created test job resulted in a working environment - containing submodules. All affted guys can verify the behaviour with my sample instructions repository https://github.com/mwiora/awx-sample-repo-w-submodules

[AlanCoding]

Thanks for the info. I'm going to put this in needs_test, because we should pursue testing with something exactly like your example.

[kdelee]

@squidboylan this looks like something you might enjoy setting up

[AlanCoding]

AWX 8.0.0 is out now. For any of the users who saw permission denied errors from the submodule clone, if you upgrade does this resolve it?

[squidboylan]

I have confirmed both the relative submodule and submodules requiring authentication work. I tested the submodule authentication with a repo that is public with a private submodule and a repo that is private with a private submodule.

[alancivita]

is there a possibility that this fix generated the bug i've described on issue #5248 ?
https://github.com/ansible/awx/issues/5248

i had to roll back to 6.1.0 to have a usable environment
thanks

[nicolaibaralmueller]

Why was this closed?

[squidboylan]

Why was this closed?

@nicolaibaralmueller I tested devel after https://github.com/ansible/awx/pull/4735 went in and the patch appears to be working, @mwiora reports the same above ( https://github.com/ansible/awx/issues/4703#issuecomment-537040707 ). Are you still experiencing this on latest AWX? What version of AWX are you on? Can you provide any logs or error messages?

[alancivita]

@nicolaibaralmueller maybe you are facing the same issue i'm facing with nasted submodules as described in #5248
Do you have submodules with nasted submodules?
apparently is a known issue not fixed yet

[rangsan]

I still facing this issue.
AWX 9.0.1
Ansible 2.9.0

Project sync with success status but when run template, it failed with below error.

Traceback (most recent call last): File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/git/objects/submodule/base.py", line 1180, in iter_items sm = rt[p] File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/git/objects/tree.py", line 298, in __getitem__ return self.join(item) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/git/objects/tree.py", line 244, in join raise KeyError(msg % file) KeyError: "Blob or Tree named 'charts' not found" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/git/objects/submodule/base.py", line 1184, in iter_items entry = index.entries[index.entry_key(p, 0)] KeyError: ('charts', 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/awx/main/tasks.py", line 1227, in run self.pre_run_hook(self.instance, private_data_dir) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/awx/main/tasks.py", line 1827, in pre_run_hook job.project.scm_type, job_revision File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/awx/main/tasks.py", line 2206, in make_local_copy for submodule in git_repo.submodules: File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/git/repo/base.py", line 324, in submodules return Submodule.list_items(self) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/git/util.py", line 934, in list_items out_list.extend(cls.iter_items(repo, *args, **kwargs)) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/git/objects/submodule/base.py", line 1188, in iter_items "Gitmodule path %r did not exist in revision of parent commit %s" % (p, parent_commit)) git.exc.InvalidGitRepositoryError: Gitmodule path 'charts' did not exist in revision of parent commit HEAD

[AlanCoding]

@rangsan given the location the error is coming from, your error does not like to me like either this issue or the nested submodules issue. Could you give me an example of an git repo that triggers this error? Or, alternatively, could you give git submodule command that will create a repo that triggers this? I'm looking for any difference from the other relative submodule test case, which is https://github.com/ansible/test-playbooks/pull/85