Setuptools: Updating MANIFEST.in does not correctly update the package sdist creates

[ghost]

Originally reported by: spookylukey (Bitbucket: spookylukey, GitHub: spookylukey)

---

The behaviour of sdist depends on previous contents of MANIFEST.in, not just the current. This is not fixed even by running setup.py clean or setup.py clean --all (although this should not be necessary).

This is very surprising behaviour, and potentially dangerous too - if someone accidentally adds a MANIFEST.in rule that includes a file that must not be distributed and notice the problem, they would expect that removing the rule will remove the file, but it does not.

I've attached a bash script that demonstrates the problem.

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/436

[miccoli]

I have been hit by this same bug. My current remedy is to delete the *.egg-info directory before running setup.py sdist.

[miccoli]

The problema appears to be in the logic with which the SOURCES.txt file is updated by python setup.py egg_info:

$ python setup.py egg_info
running egg_info
writing sample.egg-info/PKG-INFO
writing dependency_links to sample.egg-info/dependency_links.txt
writing entry points to sample.egg-info/entry_points.txt
writing requirements to sample.egg-info/requires.txt
writing top-level names to sample.egg-info/top_level.txt
reading manifest file 'sample.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sample.egg-info/SOURCES.txt'

Files already in SOURCES.txt are preserved across subsequent runs of python setup.py egg_info. Could please one of the developers clarify if this is intended behaviour and why?

[miccoli]

See also https://github.com/pypa/setuptools/blob/d4c215a7c61fb1f94b88bd2aa0b332ebaff18193/setuptools/command/egg_info.py#L560-L570
where the actual reading of SOURCES.txt takes place:

        rcfiles = list(walk_revctrl())
        if rcfiles:
            self.filelist.extend(rcfiles)
        elif os.path.exists(self.manifest):
            self.read_manifest()

I cannot understand the reason for this logic.

[miccoli]

@jaraco I think I nailed down this old (annoying bug), but nobody cares of these old bugs. Should I reopen or provide a pull request?

[jaraco]

@miccoli: If this ticket describes the issue, it's still open. A PR would be most appreciated.

[miccoli]

@jaraco yes: this ticket is accurate. Since there is no discussion active, I assume that I should provide my own PR.

[PiDelport]

I think this change may have broken sdist installs with package data: #1016

[miccoli]

The reason for

        rcfiles = list(walk_revctrl())
        if rcfiles:
            self.filelist.extend(rcfiles)
        elif os.path.exists(self.manifest):
            self.read_manifest()

is now clear: in the develop tree (under SCM control) the files to be installed are determined from the SCM system (walk_revctrl()); in an sdist install the files are to be determined from the existing manifest file, which was previously generated in the develop tree + setuptools_scm

This logic is broken for packages that do not use setuptools_scm, for which there is no guarantee that the current manifest file (in the development tree) is correct.

Unfortunately my PR #1014 badly brokes this intended behaviour, so that pip install of sdist packages under SCM control and include_package_data=True is not more possible, see #1016 .

[Firenze11]

Is there a fix to this now? Even if I delete <package>.egg_info, w hen I update MANIFEST.in and run python setup.py sdist, a new <package>.egg_info was created and the content of <package>.egg_info/SOURCES.txt was still not updated.

[miccoli]

@Firenze11 unfortunately my previous attempt at solving this bug (PR #1014) was catastrophic. (I'm still a little ashamed of the mess I made.)

But from my analysis the only place where previous content of MANIFEST.in is persisted is <package>.egg_info/SOURCES.txt. In my use cases, if you delete <package>.egg_info/SOURCES.txt and run python setup.py egg_info you get a brand-new SOURCES.txt file which is correctly populated, following the usual setuptools logic and the current MANIFEST.in.

Can you please provide an example in which SOURCES.txt is first deleted and when recreated still contains files included in a previous MANIFEST.in?

[rsyring]

I really think SOURCES.txt should be cleared automatically at some point. This is very surprising behavior. I've been trying to figure out why the changes I was making to MANIFEST.in didn't seem to be working.

[lucacerone]

Hi I am running into the same issue, but I haven't quite figured out
how to get only the code in the package "src" folder be added to the final package.

@miccoli @rsyring could you help me with a bit more explanations? I don't have a MANIFEST.in file and I am not quite sure what steps I should follow to have python setup.py sdist create the package only with the desired files.

Thanks a lot for the help!

[miccoli]

@lucacerone This issue is specific to a setup with MANIFEST.in: please first check the docs at https://packaging.python.org/guides/using-manifest-in/ and see if following those instruction you are able to obtain the desired source distribution.

The only thing that you should be aware, regarding this specific bug, is that after each change to MANIFEST.in you should delete the <package>.egg-info directory to be sure that the SOURCES.txt is regenerated appropriately.

[lucacerone]

Thanks a lot @miccoli, I managed to get the "data" in my package by adding a Manifest.in.

I don't quite understand what happens to "data" that resides outside of the "package" folder (I know it's not the preferred way, and in the end I moved the folder within the package), because if I add it to the Manifest.in, in the resulting archive I see those folders, but then they don't get copied to the site-packages folder (or any other folder I looked).

I think the documentation could be a bit more clear on what's going on under the hood, but in the end I found a way that works for me :)

Many thanks for your answer, I really appreciated it!

[tekumara]

@miccoli I've also encountered this without a MANIFEST.in, see the repro case in https://github.com/pypa/setuptools/issues/2347