Climatemachine.jl: Repository size is 95%+ due to documentation in the `gh-pages` branch

[ali-ramadhan]

Description

Not an issue for people who have fast internet connections but git clone [email protected]:CliMA/ClimateMachine.jl.git can take a while since the repository size is 160 MiB and growing.

Does it makes sense to deploy docs to a separate repository?

Main branch is only a 6.42 MiB download (12 MiB uncompressed)

➜  ~ git clone [email protected]:CliMA/ClimateMachine.jl.git --single-branch
Cloning into 'ClimateMachine.jl'...
remote: Enumerating objects: 34, done.
remote: Counting objects: 100% (34/34), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 27231 (delta 27), reused 20 (delta 20), pack-reused 27197
Receiving objects: 100% (27231/27231), 6.42 MiB | 12.60 MiB/s, done.
Resolving deltas: 100% (18268/18268), done.
➜  ~ du -hs ClimateMachine.jl 
12M     ClimateMachine.jl

but the gh-pages branch is a 154.20 MiB download (1.9 GiB uncompressed)

➜  ~ git clone [email protected]:CliMA/ClimateMachine.jl.git --single-branch --branch=gh-pages
Cloning into 'ClimateMachine.jl'...
remote: Enumerating objects: 86, done.
remote: Counting objects: 100% (86/86), done.
remote: Compressing objects: 100% (44/44), done.
remote: Total 162178 (delta 68), reused 42 (delta 42), pack-reused 162092
Receiving objects: 100% (162178/162178), 154.20 MiB | 25.30 MiB/s, done.
Resolving deltas: 100% (78764/78764), done.
Checking out files: 100% (27129/27129), done.
➜  ~ du -hs ClimateMachine.jl 
1.9G    ClimateMachine.jl

https://stackoverflow.com/a/42544963 can be used to find the responsible files but generally it's just an accumulation of many doc builds (especially if you save push previews).

[simonbyrne]

In the past I've manually gone through and deleted the previews from the history, but it would be nice to have an automated way to do this.

[simonbyrne]

FWIW, I cleaned up the history using the following:

git checkout gh-pages
mkdir previews_new
gh pr list --limit=1000 | cut -f1 | while read N; do mv "previews/PR$N" previews_new; done
git rm -rf previews
mv previews_new previews
git commit -m "remove old previews"
git branch gh-pages-new $(echo "delete history" | git commit-tree HEAD^{tree})
git push --force origin gh-pages-new:gh-pages

[kpamnany]

Seems like a GitHub Action could be set up to do this after pushes to master? I'm a bit amazed that I can't find anything about this problem...

[charleskawczynski]

I'm a bit amazed that I can't find anything about this problem...

I was thinking the same thing after searching around for a bit

[simonbyrne]

we could set up a cron job?

[simonbyrne]

there is also an option to add a hook after a PR is closed.

[simonbyrne]

perhaps something like this?:

on:
  pull_request:
    types: [closed]

jobs:
  doc-history-cleanup:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout gh-pages branch
        uses: actions/checkout@v2
        with:
          ref: gh-pages

      - name: Delete preview and history
        run: |
            git rm -rf "previews/PR$PRNUM"
            git commit -m "delete preview"
            git branch gh-pages-new $(echo "delete history" | git commit-tree HEAD^{tree})
        env:
            PRNUM: ${{ github.event.number }}

      - name: Push changes
        run: |
            git push --force origin gh-pages-new:gh-pages

[charleskawczynski]

So, this will delete a single preview that corresponds to the closed PR?

[simonbyrne]

and delete the github pages history

[charleskawczynski]

Does that mean that previews from other PRs will be deleted?

[simonbyrne]

No, it deletes the preview corresponding to the PR that triggered the build, and then resets the git history on that branch; the contents stay the same.

[charleskawczynski]

Sounds good to me!

[simonbyrne]

https://github.com/CliMA/TimeMachine.jl/blob/master/.github/workflows/DocCleanup.yml seems to work.