Gitpython: Bug: Getting tree on same file in different commits

[crazyhouse33]

Hi, I am observing something weird using your library version 3.1.7 (Debian + python3.7/3.5)

Context:
I am creating a little library to explore files in commits by specifying a path relative to the normal python environment (cwd and whole system). The test failing is the following:

generator = gfe.explore_commits(gfe.get_repo('test_gfe_disapearring_file'), reverse=True) # Setting up generator based on your repo object
commit = next(generator)
#In this commit test2 does not exist
files = commit.search_files(['test_gfe_disapearring_file/test', 'test_gfe_disapearring_file/test2'])
assert len(files)==1

#In this commit it does
 commit = next(generator)
 files = commit.search_files(['test_gfe_disapearring_file/test', 'test_gfe_disapearring_file/test2'])
 assert len(files)==2

The first call to my function work as expected, the second have weird behavior, see the following debugging section:

res.append((path, (self.tree / from_repo_path).data_stream)) # This is the problematic code in my library, second call of previous piece of code, on the existing file
(Pdb) p self.tree
*** ValueError: SHA could not be resolved, git returned: b''
(Pdb) p self.tree
*** IndexError: list index out of range
(Pdb) p self.tree
<git.Tree "149b75cfadaf67444642e1ff0e5c009ba2829526">
(Pdb) p self.tree/from_repo_path
*** IndexError: index out of range
(Pdb) p self.tree/from_repo_path
*** IndexError: index out of range
(Pdb) p self.tree/from_repo_path
<git.Blob "fbcf12d50552354fc878706bacea89fbb3f9f999">
p (self.tree/from_repo_path).data_stream
(b'\x14\x9bu\xcf\xad\xafgDFB\xe1\xff\x0e\\\x00\x9b\xa2\x82\x95&', b'tree', 32, <git.cmd.Git.CatFileContentStream object at 0x7f9fcf864f48>)

Notice how I cant access the commit.tree the first time (Value Error) or the second time (Index Error), but then suddenly I can. In a similar way the / operator emit an Index error 2 times before working.

Meanwhile I will look for some dirty fix in my code to avoid the issue, I would appreciate very much your insight on this. This look like an unexpected state related issue while accessing same path on different commits.

Here is the code of the used functions:

# This is my wrapper around gitpython
# we add a convenient method to gitpython commit
def search_files(self, paths):
    """Search files relative to cwd in commit, return a list of (path,stream)"""
    res=[]
    if type(paths)==str:
        paths= [paths]
    for path in paths:
        from_repo_path=os.path.relpath(path,start=self.repo.working_dir)#gitpython only work with path relative to repo 
        try:
            res.append((path, (self.tree / from_repo_path).data_stream))
        except KeyError: #Not existing in the commit
            pass
    return res

git.objects.Commit.search_files = search_files

def explore_commits(repo, paths_restriction=[], branchs=None,
        commits=None, filter=None, **kwargs):
    """Generator containing commits matching your selections. Branch is a space separated list of branch. current branch by default. Kwargs is argument you can give to the git rev-list command, such as reversed=True, filter=[], skip=5, after...
    Filter is a list of begins of hash. Any commit having same begining will be skipped
    """
    if filter is None:
        filter = []

    root = repo.working_dir
    paths = [os.path.relpath(path, start=root) for path in paths_restriction]
    if commits is None:
        try:
            commits = [
                    commit for commit in repo.iter_commits(
                        paths=paths_restriction,
                        rev=branchs,
                        **kwargs)]
        except ValueError:  # No commits
            commits = []

    else:
        commits = [repo.rev_parse(commit) for commit in commits]
    for commit in commits:
        if not any(commit.hexsha.startswith(filter_comm)
                for filter_comm in filter):
            yield commit

Thanks for your the lib btw.

[Byron]

I am so sorry for this one - it's a known footgun that is utterly undocumented and to my mind tradeoff that should not have been made: performance over correctness.

By default, GitPython uses the GitCmdObjectDB, which keeps around two git-cat-file processes to provide object information. These resources are inherently shared. The code in the issue seems to be storing streams for later, which can cause all kinds of side-effects. If I remember correctly, when the stream is created it reads a little from the process, and expects the stream to be consumed/read to end for this to work.

There are two workarounds.
First one could use a pure python GitDB implementation when instantiating a repository. This is much slower to use and is even less suited for long-running processes, but won't exhibit this particular issue.

Alternatively one could read the streams and store the resulting object - most objects are small so that shouldn't be an issue.

Does this help? If so, please feel free to close this issue.

[crazyhouse33]

Hi Byron. I will try to avoid the problem by putting the content of the stream in another handcrafted stream. When I am done with it I will confirm you if it's working and if so publish here my workaround, then close the issue. Maybe we should change the name of the issue as well with a better name so people than run into this bug see it quickly.

Thanks for your confirmation as it allow me to be one hundred percent sure that it's a bug on your side.
Good luck with your project.

[crazyhouse33]

Okay this was my way to deal with it.

from io import BytesIO
from shutil import copyfileobj

def get_stream_cpy(datastream):
    """Take a binary stream like object and return a file like copy with same content, needed because of https://github.com/gitpython-developers/GitPython/issues/1070"""
    res= BytesIO()
    copyfileobj(datastream, res)
    res.seek(0)
    return res

def search_files(self, paths):
    """Search files relative to cwd in commit, return a list of (path,stream)"""
    res=[]
    if type(paths)==str:
        paths= [paths]
    for path in paths:
        from_repo_path=os.path.relpath(path,start=self.repo.working_dir)#gitpython only work with path relative to repo 
        #Needed because of https://github.com/gitpython-developers/GitPython/issues/1070

        try:
            datastream= (self.tree/from_repo_path).data_stream
        except KeyError:#File not existing in the commit
            continue
        res.append((path, get_stream_cpy(datastream)))
    return res

Everything work fine for me now as far as I can tell, I let you change the name of the issue as you thinks its best suited.