Ldaprecord-laravel: [Feature] decrease resource usage on large user imports

[aanderse]

Describe the feature you'd like:

Hi @stevebauman :wave: I was wondering if you have looked at possible optimizations for the user import command. With a data set of 20k+ users I need to allocate about 2G of ram for the command to run and the import takes a few minutes. I was planning on looking at possible ways to decrease ram usage but figured I would ask if this is something you have put any time/effort into before I start.

As always I appreciate any feedback.

[stevebauman]

Hi @aanderse! Great question.

There's a couple ways you can optimize memory usage -- I'll walk you through both. You can also combine these two ways to minimize resource usage.

1. Break the imports into chunks:

Since you have a lot of users that you are importing, it may help to break them into smaller chunks and process each chunk individually.

To do this, you create your own command and make a range() of letters to import users that start with each letter, and then call ldap:import with the --filter option:

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;

class LdapChunkImport extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'ldap:chunk-import';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Import LDAP users by chunks.';

    /**
     * Execute the console command.
     *
     * @return mixed
     */
    public function handle()
    {
        foreach (range('a', 'z') as $letter) {
            $this->call('ldap:import', [
                'provider' => 'ldap',
                '--no-interaction',
                '--restore' => true,
                '--delete' => true,
                '--filter' => "(cn=$letter*)",
            ]);
        }
    }
}

1. Select specific attributes:

Selecting specific attributes to be returned by your LDAP server will greatly reduce your memory usage. By default, all attributes will be returned from your LDAP server, so this is a good step in optimization.

To do this right now, you will have to extend the current import command and override a method:

<?php

namespace App\Console\Commands;

use LdapRecord\Laravel\LdapUserRepository;
use LdapRecord\Laravel\Commands\ImportLdapUsers as BaseCommand;

class ImportLdapUsers extends BaseCommand
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'ldap:custom-import';

    /**
     * The attributes to select.
     * 
     * @var array
     */
    protected $attributes = ['cn', 'mail', 'samccountname'];

    /**
     * Retrieves users to be imported.
     *
     * @param LdapUserRepository $users
     *
     * @return \LdapRecord\Models\Model[]
     */
    public function getUsers(LdapUserRepository $users)
    {
        $query = $users->query();

        // Select a limited number of attributes.
        $query->select($this->attributes);

        if ($filter = $this->option('filter')) {
            // If the filter option was given, we'll
            // insert it into our search query.
            $query->rawFilter($filter);
        }

        if ($user = $this->argument('user')) {
            return [$query->findByAnr($user)];
        }

        // Retrieve all users. We'll paginate our search in case we
        // hit the 1000 record hard limit of active directory.
        return $query->paginate()->toArray();
    }
}

I'm going to patch this so you can include selected attributes in the command itself to minimize memory usage, and maybe include a range option as well so you don't have to do this yourself.

I'll leave this issue open until it has been patched in. Hope this helps!

[aanderse]

That looks pretty good. Thanks. Would the chunking work with soft deletes, though? This query specifically has me concerned:

        $deleted = $eloquent->newQuery()
            ->whereNotNull($eloquent->getLdapGuidColumn())
            ->where($eloquent->getLdapDomainColumn(), '=', $domain)
            ->whereNotIn($eloquent->getLdapGuidColumn(), $this->imported)
            ->update([$eloquent->getDeletedAtColumn() => $deletedAt = now()]);

[stevebauman]

Whoops, good catch! You do not want to use the --delete-missing flag with the chunking solution. I've removed it from my example in case someone else looks at it in the future. All other flags however can be used.

[aanderse]

Selectively pulling attributes cut my memory usage to a third :+1: As always, @stevebauman delivers :smile:

[stevebauman]

Awesome! That's great -- haha thanks @aanderse, glad I was able to help :smile:. Patches are incoming for the import command. 👍

[stevebauman]

These changes were released in v1.7.0. We're all set!

[aanderse]

Thanks again @stevebauman!