Odo: Re-enable support for the sparseCheckoutDir field in devfiles

[johnmcollier]

/kind bug
/area devfile
/area regression

What versions of software are you using?

Operating System:
macOS

Output of odo version:
master

How did you run odo exactly?

odo create java-openliberty --starter

Actual behavior

When cloning a starter project with the sparseCheckoutDir field set, odo create --starter now clones the entire source code repo specified in the starter project, rather than checking out the directory specified in sparseCheckoutDir.

As the java-openliberty devfile currently uses sparseCheckoutDir for its starterProject, we're currently unable to use odo create java-openliberty --starter and successfully deploy a component from it.

Expected behavior

odo create --starter respects the sparseCheckoutDir field in devfiles

Any logs, error output, etc?

Sample devfile:

schemaVersion: 2.0.0
metadata:
  name: nodejs
  version: 1.0.0
starterProjects:
  - name: nodejs-starter
    git:
      remotes:
        origin: "https://github.com/odo-devfiles/nodejs-ex.git"
      sparseCheckoutDir: /test
components:
  - name: runtime
    container:
      image: registry.access.redhat.com/ubi8/nodejs-12:1-45
      memoryLimit: 1024Mi
      mountSources: true
      sourceMapping: /project
      endpoints:
        - name: http-3000
          targetPort: 3000
commands:
  - id: install
    exec:
      component: runtime
      commandLine: npm install
      workingDir: /project
      group:
        kind: build
        isDefault: true
  - id: run
    exec:
      component: runtime
      commandLine: npm start
      workingDir: /project
      group:
        kind: run
        isDefault: true

[adisky]

it has been discussed here https://github.com/openshift/odo/pull/3867#discussion_r483576554

[johnmcollier]

Thanks @adisky. I'll go update the issue title since its disabling was deliberate

In that case I think we need to do two things:
1) Update the java-openliberty devfile to not rely on the sparseCheckoutDir
2) (in the future) Re-introduce support for the sparseCheckoutDir

Have we looked at how Che handles the sparseCheckoutDir field? Maybe they have a better approach than what we were doing.

CC @kadel

[elsony]

Disabling that will be a problem with the java-openliberty stack. @scottkurz @ajm01 This means that the liberty starter scenario will not work out of the box as before. You may consider to adjust the java-openliberty stack to workaround this problem for the time being.

[johnmcollier]

+1 on updating the stack to workaround it for now. From Tomas' comment on the PR, it sounds like we weren't handling sparseCheckoutDir in a good way, so until we find a good way to get it working, we should make sure the java-openliberty stack is fixed at least

If it may take a while to re-introduce the sparseCheckoutDir functionality, we may also want to consider adding a warning message to odo create --starter when sparseCheckoutDir is present in a devfile

[scottkurz]

@johnmcollier @elsony - How would you suggest enabling the ability to have multiple starters?

It would be easy enough to switch to a new repository with a single starter and nothing else, but we intended to offer:

odo create java-openliberty --starter
odo create java-openliberty --starter OtherStarterProj

Can each individual starter get its own relevant config in the devfile?

(I tried it quick on something close to the odo2 beta and it always checks out the repo root no matter what the --starter arg).

[johnmcollier]

@scottkurz So, devfiles can have multiple starter projects and your syntax (--starter <name>) is correct. For example:

starterProjects:
  - name: nodejs-starter
    git:
      remotes:
        origin: "https://github.com/odo-devfiles/nodejs-ex.git"
  - name: second-starter
    git:
      remotes:
        origin: "https://github.com/johnmcollier/nodejs-ex.git"
      sparseCheckoutDir: /test

odo create nodejs --starter nodejs-starter would checkout the entire repo and odo create nodejs --starter second-starter would checkout the test folder from https://github.com/johnmcollier/nodejs-ex.git (if the sparseCheckoutDir field was working)

[elsony]

@scottkurz Just to be clear, to work around the current problem of sparseCheckoutDir not supported yet. You can still have multiple starter projects. For those different starter projects, you either:

1. put the starter code under different repos and reference the starter projects using different repo
2. put them under different branches of the same repo and use the revision in checkoutFrom to control what to checkout for each of the starter project (see https://github.com/devfile/api/pull/120#issue-470846182 for example of the syntax)

[kadel]

Thanks @adisky. I'll go update the issue title since its disabling was deliberate

In that case I think we need to do two things:

1. Update the java-openliberty devfile to not rely on the sparseCheckoutDir
2. (in the future) Re-introduce support for the sparseCheckoutDir

Have we looked at how Che handles the sparseCheckoutDir field? Maybe they have a better approach than what we were doing.

CC @kadel

@johnmcollier, As far as I know, they don't handle that, as they don't have v2 support and this was not in v1 :-/
One solution for this could be to just clone the whole repo to a temporary directory and then copy just sparseCheckoutDir. This would work for starterProject but not for project as in that case, the directory should stay as a cloned git repo. And from what I read about sparse checkout it doesn't work like this.

Sparse checkout in git can populate only certain directories, but the directory structure remains.

For example, If I have a repo with two dirs /dir1/file1.txt and /dir2/file2.txt and I want to do sparse checkout if a /dir2 as git does it would end up with a directory with /dir2/file2.txt.
But what we want to do here is to end up with /file2.txt and that is not what sparse checkout does.

I would actually consider completely removing this field from Devfile, It will create a lot of confusion and problems.

Created https://github.com/devfile/api/issues/128

[kadel]

• put the starter code under different repos and reference the starter projects using different repo
• put them under different branches of the same repo and use the revision in checkoutFrom to control what to checkout for each of the starter project (see devfile/api#120 (comment) for example of the syntax)

revision is not fully implemented yet :-(

The only way how we can fix it right now is to go with option 1 (separate repo)

[elsony]

@scottkurz ^^ We only have option 1 for the time being.

[dev-gaur]

From what I could infer from the above discussion,

sparseCheckoutDir is being replaced by the subDir property. And the subDir property would try to comply with how git sparse checkout works ( https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/ )

@kadel @girishramnani @johnmcollier @elsony can you reassure on this? please correct me if I'm wrong.

[kadel]

sparseCheckoutDir is being replaced by the subDir property. And the subDir property would try to comply with how git sparse checkout works ( https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/ )

@kadel @girishramnani @johnmcollier @elsony can you reassure on this? please correct me if I'm wrong.

yes to first part no to second part :-)
It will be renamed to subDir because it won't comply with how git sparse checkout works.

[dev-gaur]

Some more clarity now after a discussion with @kadel .. subDir property would simply clone the specified sub-directory on the project's base directory path.

Waiting on https://github.com/devfile/api/pull/129 before moving ahead with this.

[mik-dass]

A function handling sparseCheckoutDir is already present in the master branch https://github.com/openshift/odo/blob/4bb02182db17c445ab5762d97486a7ca5aee886c/pkg/odo/cli/component/create.go#L1178 but currently is only used for zip based starterProjects

[mik-dass]

For git component, if a subDir is set, we will clone the entire directory to a temporary directory and copy the files inside the subDir directory and place it in the component directory.