Zipline: Add options to zipline clean

[freddiev4]

Was playing around with the zipline clean command to get rid of old bundle data and was thinking it would be nice to have two other options/ways of cleaning

• zipline clean --all: which will just get rid of every data bundle you've ingested regardless of time
• zipline clean with a --before, --after, and --keep-last option: if you don't specify the bundle, it will just run that command for every bundle.
  
  
  
  • Example 1: zipline clean --before 2018-01-01 will just clean every bundle before that date, for every registered bundle
  
  
  • Example 2: zipline clean --keep-last 1 will keep the the most recent run of every registered bundle and remove all the others.
  
  

Thoughts @llllllllll et all?

[llllllllll]

I like both of these a lot.

[ssanderson]

:+1:

[freddiev4]

For anyone who wants to get started with this, the code lives in zipline/data/bundles/core.py in def clean().

Resources:

• Zipline Data Bundle Docs
• Development Guidelines
• def clean() code below:

    @preprocess(
        before=optionally(ensure_timestamp),
        after=optionally(ensure_timestamp),
    )
    def clean(name,
              before=None,
              after=None,
              keep_last=None,
              environ=os.environ):
        """Clean up data that was created with ``ingest`` or
        ``$ python -m zipline ingest``

        Parameters
        ----------
        name : str
            The name of the bundle to remove data for.
        before : datetime, optional
            Remove data ingested before this date.
            This argument is mutually exclusive with: keep_last
        after : datetime, optional
            Remove data ingested after this date.
            This argument is mutually exclusive with: keep_last
        keep_last : int, optional
            Remove all but the last ``keep_last`` ingestions.
            This argument is mutually exclusive with:
              before
              after
        environ : mapping, optional
            The environment variables. Defaults of os.environ.

        Returns
        -------
        cleaned : set[str]
            The names of the runs that were removed.

        Raises
        ------
        BadClean
            Raised when ``before`` and or ``after`` are passed with
            ``keep_last``. This is a subclass of ``ValueError``.
        """
        try:
            all_runs = sorted(
                filter(
                    complement(pth.hidden),
                    os.listdir(pth.data_path([name], environ=environ)),
                ),
                key=from_bundle_ingest_dirname,
            )
        except OSError as e:
            if e.errno != errno.ENOENT:
                raise
            raise UnknownBundle(name)
        if ((before is not None or after is not None) and
                keep_last is not None):
            raise BadClean(before, after, keep_last)

        if keep_last is None:
            def should_clean(name):
                dt = from_bundle_ingest_dirname(name)
                return (
                    (before is not None and dt < before) or
                    (after is not None and dt > after)
                )

        elif keep_last >= 0:
            last_n_dts = set(take(keep_last, reversed(all_runs)))

            def should_clean(name):
                return name not in last_n_dts
        else:
            raise BadClean(before, after, keep_last)

        cleaned = set()
        for run in all_runs:
            if should_clean(run):
                path = pth.data_path([name, run], environ=environ)
                shutil.rmtree(path)
                cleaned.add(path)

        return cleaned