Gulp: copy and keep original folder structure

[pajooh]

i have a list of files to copy:

    gulp.src([list of js files])
      .pipe(gulp.dest(build_dir))

the files are copied, but flattened.
how can i reserve original directory structure? i can't find an option for that

[contra]

Can you paste the globs you are using?

[pajooh]

i use external config.json:

{
  "build_dir": "build",
  "vendor_files": {
    "js": [
      "vendor/angular/angular.js",
      "vendor/angular-animate/angular-animate.js",
      "vendor/angular-bootstrap/dist/ui-bootstrap-tpls-0.7.0.min.js",
      "vendor/angular-ui-router/release/angular-ui-router.js",
      "vendor/angular-ui-utils/modules/route/route.js",
      "vendor/restangular/dist/restangular.js",
      "vendor/underscore/underscore.js",
      "vendor/angular-underscore/angular-underscore.js"
    ],
    "css": [
    ],
    "assets": [
    ]
  }
}

and gulpfile.js

var config = require('./build.config.json');
gulp.task('scripts', function(){
    gulp.src(config.vendor_files.js)
      .pipe(gulp.dest(config.build_dir + '/scripts/vendor'))
});

[contra]

@pajooh what do you want the files coming out to look like?

[pajooh]

i want the original paths, i.e.
"vendor/angular/angular.js",
"vendor/angular-animate/angular-animate.js",
"vendor/angular-bootstrap/dist/ui-bootstrap-tpls-0.7.0.min.js",
"vendor/angular-ui-router/release/angular-ui-router.js",
"vendor/angular-ui-utils/modules/route/route.js",
"vendor/restangular/dist/restangular.js",
"vendor/underscore/underscore.js",
"vendor/angular-underscore/angular-underscore.js"
under build directory, now all js files are coppied to the top level directory

[phated]

It seems that gulp only preserves the directory structure of globs. @Contra is this correct?

@pajooh is there a reason globs aren't being used?

[pajooh]

i have some bower installed packages, with a bunch of js files in them.
i don't need them all, but some specific files(e.g. just minified versions).
i have to list them one by one, globs can not help here

[phated]

@pajooh you can still use globs

An example being:

{
  "build_dir": "build",
  "vendor_files": {
    "js": [
      "**/angular.js",
      "**/angular-animate.js",
      "**/ui-bootstrap-tpls-0.7.0.min.js",
      "**/angular-ui-router.js",
      "**/route.js",
      "**/restangular.js",
      "**/underscore.js",
      "**/angular-underscore.js"
    ],
    "css": [
    ],
    "assets": [
    ]
  }
}

but you could make it more precise. It seems like such a pain to type out the whole path and keep that up-to-date.

[pajooh]

yes, but the glob approach is not generally the best way.
think if some js exist in many folders with the same name, glob approach may get worse than typing out the whole path

[phated]

@pajooh that's why I said you can get more specific.

The best approach would be to not expect that directory structure in your output, since that is vendor code anyway.

[pajooh]

lets see an example
i want to use jquery.js from jquery package, if i use **/jquery.js, its too general, because some other packages may already has that, the proper glob seems to be jquery/**/jquery.js,
it will do the job, but the folder structure does not contain top jquery folder!
so i think there is a need for gulp option to preserves the directory structure of non globs

[stryju]

temporary solution that worked for me just fine:

{
  "build_dir": "build",
  "vendor_files": {
    "js": [
      "angular/angular.js",
      "angular-animate/angular-animate.js",
      "angular-bootstrap/dist/ui-bootstrap-tpls-0.7.0.min.js",
      "angular-ui-router/release/angular-ui-router.js",
      "angular-ui-utils/modules/route/route.js",
      "restangular/dist/restangular.js",
      "underscore/underscore.js",
      "angular-underscore/angular-underscore.js"
    ],
    "css": [
    ],
    "assets": [
    ]
  }
}

// gulpfile
var config = require('./build.config.json');
gulp.task('scripts', function(){
    gulp.src(config.vendor_files.js, { cwd : 'vendor/**' })
      .pipe(gulp.dest(config.build_dir + '/scripts/vendor'))
});

---

It seems that gulp only preserves the directory structure of globs

@phaet seems like it - is that unexpected?

[pajooh]

@stryju very good, your temporary solution worked for me
i used:

// gulpfile
var config = require('./build.config.json');
gulp.task('scripts', function(){
    gulp.src(config.vendor_files.js, { cwd : '**/vendor' })
      .pipe(gulp.dest(config.build_dir + '/scripts'))
});

[stryju]

@pajooh glad i could help

now, the question remains - is that glob behavior a desired behavior?
the only difference is the base prop on the streams

[contra]

Yup - gulp gets the output file structure from the relativity between the base path (where the glob ended) and the cwd

[contra]

When you give it a direct file it won't maintain the structure. To keep the structure you can pass in a base arg to src or you can use gulp-rename, gulp-flatten, etc. to mess with the structure

[steida]

I made a example for side by side compilation. Maybe useful as recipe: https://gist.github.com/steida/8592161.

[dwelch2344]

The solutions here didn't work in my case (specifically copying only the angular source map to a flat folder) so I wrote this simple plugin. Hope it can help someone out:
https://gist.github.com/dwelch2344/8969706

[contra]

@dwelch2344 Isn't that basically what gulp-flatten does?

[dwelch2344]

Ah, indeed it does, though not nearly as pretty. We'll be switching to this. Thanks for the heads up!—
Sent from Mailbox for iPhone

On Wed, Feb 12, 2014 at 11:40 PM, Eric Schoffstall
[email protected] wrote:

@dwelch2344 Isn't that basically what gulp-flatten does?

Reply to this email directly or view it on GitHub:
https://github.com/gulpjs/gulp/issues/151#issuecomment-34952423

[dminkovsky]

Long thread here for a very simple solution.

Just set the files' file.base so that their file.relative resolves to the path you want for each file relative to the build dir (dest). That's all.

In any such situation, if you can't set the base with config, you can effect such a transformation with a through-like stream right in your task, instead of using a gulp- module. Like

// [ ... ]
.pipe(through(function(file) { file.base = yourBase; this.emit('data', file); }))
// [ ... ]

All if this, by the way, is behavior directly from require('vinyl-fs').src and require('vinyl-fs').dest, because require('gulp').src and require('gulp').dest are identically those objects.

[contra]

We are allowing functions in gulp.dest so you could do this

gulp.dest(function(file){
  return path.join(build_dir, path.dirname(file.path));
});

Track https://github.com/wearefractal/vinyl-fs/pull/15

[TheSisb]

@contra: I don't know why but whenever I try that I keep getting

gulp] 'compress' errored after 4.38 ms Invalid output folder

I tried many variations of the same command.
I even tried switching it to simply: .pipe(gulp.dest(function(file){ return '../js'; }); and still get the same error.

[contra]

@TheSisb I linked the open issue. It hasn't been merged yet.

[jbdemonte]

Hi, where did you saw that dist may take a function as argument?
it inherit vinyl-fs, and its code contains (https://github.com/wearefractal/vinyl-fs/blob/master/lib/dest/index.js)

  if (typeof outFolder !== 'string') throw new Error('Invalid output folder');

[jbdemonte]

If you want to keep the sub-folders, you have to define the base in the options (2nd argument)
e.g., to keep "assets/file.doc" in "dist/":

    gulp.src(["assets/file.doc"], {base: "."})
        .pipe(gulp.dest("dist/"));

[contra]

@jbdemonte https://github.com/wearefractal/vinyl-fs/pull/15

[jbdemonte]

:) I missed that, but, using the base option works fine

[nmccready]

@jbdemonte I love you on base!

[choffmeister]

@jbdemonte Thanks I really wasted some time on this problem until you gave the solution!

[andreasonny83]

This will copy the file from inside the _temp folder into the dist folder keeping the directory structure:

gulp.task('copy_to_dist', function() {
gulp.src([
'index.html',
'_temp/css//*',
'_temp/img//_',
'_temp/js/__/_'
])
.pipe(gulp.dest(function(file) {
var dest = file.base.replace('_temp', 'dist');
return dest;
}));
});

[DanielD]

I used this:

gulp.task("copy-assets", function() { var assets = { js: [ paths.npm + "es6-shim/es6-shim.min.js", paths.npm + "systemjs/dist/system-polyfills.js", paths.npm + "systemjs/dist/js/system.src.js", paths.npm + "bootstrap/dist/js/bootstrap.js", paths.npm + "angular2/es6/dev/src/testing/shims_for_IE.js", paths.npm + "angular2/bundles/angular2.dev.js", paths.npm + "angular2/bundles/router.dev.js", paths.npm + "angular2/bundles/angular2-polyfills.js", paths.npm + "angular2/bundles/http.dev.js", paths.npm + "rxjs/bundles/Rx.js", paths.npm + "typescript/lib/typescript.js" ], css: [ paths.npm + "bootstrap/dist/css/bootstrap.css" ] }; gulp.src(assets.js, { base: paths.npm }) .pipe(gulp.dest(paths.npmOutput)); });

where, paths.npm and paths.npmOutput are './node_modules/' and './wwwroot/node_modules/', respectively.

[slavafomin]

It's very counter-intuitive to me why Gulp flattens the filestructure when globs are not used. By the way I've tried to use base option of the src function, but it's not working. I've managed to make it work only after replacing base with cwd and using ** in it. But for some reason, the official documentation doesn't mention cwd option. I'm using Gulp 3.9.1,

[nekhaly]

Just to conclude in a simple statement :) .. Use **/ in src to keep folder structure in dest
Like this
gulp.src('**/views/*.html').pipe(gulp.dest('build'));
considering views folder in the same level with gulpfile.js

[rlvandaveer]

If you are using an array of paths as the argument to src, the folder structure will not be respected using the pattern **/, e.g. gulp.src(['media/*', 'media/norespect/**/']).pipe(gulp.dest('build')); The destination will be flattened. It does work if it's the only pattern passed to src.

[oychao]

@jbdemonte Thank you so much.

[maltem-za]

@rlvandaveer I found this SO answer to be helpful: https://stackoverflow.com/a/32880264

[dennisbaskin]

tl;dr of our issue

I needed to move one nested subdirectory and its contents out into my build directory, with a custom structure

Working Solution

This is a fairly old thread but I found @andreasonny83 's answer to be the right path for what I was trying to accomplish.

What I needed is to have specific "component" directories that contained nested structures with demos moved to my build/demos directory. I have many components in a directory named components. Not all of the component-... subdirectories needed to be moved, only a select few. I needed to be able to copy the selected few, and avoid an additional components directory being added (when using base option). I was able to accomplish this through the following as suggested by @andreasonny83 :

gulp.task('task-name', () => {
  return gulp.src([
    'components/component1/**',
    'components/component2/**',
  ]).pipe(gulp.dest((file) => {
    return file.base.replace('components', 'build/demos')
  }))
})

The following shows the before (with components) and after (with the sibling build directory) file structure:

components/
├── component1/
│   └── subdir/
├── component2/
│   └── subdir/
├── component3-no-copy/
├── component4-no-copy/
build/
└── demos/
    ├── component1/
    │   └── subdir/
    └── component2/
        └── subdir/

If someone knows a more idiomatic approach to this, it would be great to know. But for now, this is decent workaround.

[jeffreyscottfrench]

Simplified Version

This method is based off the response from @jbdemonte, and has worked for me, and also seems to be the most straightforward I've seen so far:

• define your input files/globs with paths relative to project base, either in a variable or inline in the gulp.src first parameter
• set your gulp.src base option to the source folder in the second parameter
• set your output folder as usual

Your code would look like this, where ./src is your source folder and ./dest is your destination folder:

var filesToCopy = ['./src/file.txt', './src/.hiddenFile', './src/subFolder/**/*.+(gif|png)'];

gulp.task('copyFiles'), function(){
    return gulp.src( filesToCopy, { base: './src' } )
    .pipe( gulp.dest( './dest' )
}

Hope that helps!