Faker.js: Uniqueness

[jaredpalmer]

Current experiencing issues with faker not generating unique enough values. This results in a false negative test result. Our workaround is to concatenate or prepend another faker function to increase randomness.

Notoriously not-unique-enough faker funcs for running integration tests with:

• faker.internet.ip()
• faker.internet.email()

[Marak]

@jaredpalmer -

I can confirm conflicting values for both faker.internet.ip() and faker.internet.email() when running the attached code.

It seems on master we are seeing about 3 IP conflicts per 100,000 and 700 email conflicts per 100,000.

We should for sure fix this for email address. I should review other faker methods that may also require more entropy. RE: #245

var faker = require('../../index');
faker.locale = "en";


var ips = {};
var conflicts = 0;
for (var i = 0; i < 100000; i++) {
  var ip = faker.internet.ip();
  if (typeof ips[ip] === 'undefined') {
    //console.log('set ip', ip)
    ips[ip] = ip;
  } else {
    conflicts++;
    console.log('CONFLICTS', ip)
  }
}
// 0 at 10,000
// 3 at 100,000


//console.log(faker.image.image())
var emails = {};
var conflicts = 0;
for (var i = 0; i < 100000; i++) {
  var email = faker.internet.email();
  if (typeof emails[email] === 'undefined') {
    //console.log('set email', email)
    emails[email] = email;
  } else {
    conflicts++;
    console.log('CONFLICTS', email)
  }
}
console.log('total conflicts', conflicts)

// 3 - 7 at 10,000
// 652 at 100,000

[Marak]

I've looked at the code for generating emails / usernames and I'm not 100% certain why we are seeing conflicts at this scale.

https://github.com/Marak/faker.js/blob/master/lib/internet.js#L69

We should be generating enough uniqueness here to not be seeing conflicts with such small datasets? Can anyone confirm the math based on the code conditionals and first / last name data?

If anything, we may have to implement a way for faker to remember previously generated results and then re-generate until a new unique is found? I feel like we can improve the randomness without having to resort to that.

[Marak]

RE: https://github.com/Marak/faker.js/pull/477

[agm1984]

I was working with an anti-duplicate script this weekend. I will see if I can produce you some logic along these lines. I'm thinking that faker should maintain some kind of local store of values and check against it to avoid duplicates. It could simply generate a new random every time it detects duplicate was already made in the session or lately (however the store is made to function).

In my testing this weekend, it is suitably performant to check against a growing text file. It took about 0.3ms to check against 1 entry and it took 10 seconds to generate 100,000 unique 16 digit values synchronously, according to console.time('foo') and console.timeEnd('foo')

The basic idea would be to create a psuedo-Redis local store, even better if it integrates with an existing Redis store.

The issue is it needs a working memory so it can avoid offering something it already offered, in my opinion. This will offer infinite horizontal scalability if a common store is used.

[Marak]

@agm1984 - Sounds good!

Doing a pluggable Store seems like the right idea, and should be easy to do. I have been using the following Store.js file at scale with success. It should also work as drop-in for node-redis module, see: https://github.com/Stackvana/microcule/blob/master/lib/plugins/Store.js

I see the memoization / pluggable data-store as the easier part of this problem. The harder part will be generalization of code and configurability of uniqueness API. The functionality for unique API calls should probably be general enough to work on any JavaScript method? Let me know what you figure out.

There has been a bit of discussion over the years about uniqueness and tangentially related issues. If you haven't already, I'd recommend reading this thread and checking out some of the linked issues.

Looking forward to seeing what you find. I'd also recommend deep diving into any problem faker methods you've run into in the past to see if we can make any changes to the core API itself for those methods to improve uniqueness or quality of generated data.

[agm1984]

Here is a kind of proof of concept I just made. It seems to be working perfectly for me. Feel free to use it:

https://github.com/agm1984/uniqueNameGenerator

I have to work on my main project here now lol, but I will circle back. I have a need to build a test module soon that generates fake businesses and users with a lengthy list of properties and graph DB relationships all randomizing, so I will definitely try to incorporate it into faker so I can just import faker.

I didn't want to install babel so I just rolled with this syntax. It also added 30% execution time when I used promises, so I kept it full synchronous. It was a difference of 0.3ms instead of 0.2ms per name of length 5~.

[agm1984]

The issues I had with faker, to answer your thought, were with generating Business Names, and with generating random images. I recall every method to do with images was returning the same image. It would be nice to have it pull from some free source. There should be lots of APIs for that.

If you just make a POST endpoint and a list view and dump like 10,000 records containing usage of every method, you will see where the duplication spots are :)

[Marak]

@jaredpalmer - I've done a bit of work in master to improve the situation for creating unique data. It's actually looking quite good.

We now have a new API method faker.unique to help in ensuring unique data generation from faker methods.

Check out the example

Looks something like this:

var faker = require('../../index');

var emails = {};
var conflicts = 0;
// emails estimated: 1,055,881
// full names estimated: 1,185,139
for (var i = 0; i < 100000; i++) {

  // call function with no arguments
  var email = faker.unique(faker.internet.email);

  // or with function arguments as argument array
    // var email = faker.unique(faker.internet.email, [null, null, 'marak.com']);

  // or with custom options for maxTime as milliseconds or maxRetries
    // var email = faker.unique(faker.internet.email, [null, null, 'marak.com'], { maxRetries: 1, maxTime: 50 });

  if (typeof emails[email] === 'undefined') {
    // found a unique new item
    emails[email] = true;
  } else {
    // found a conflicting item ( should not happen using faker.unique() )
    conflicts++;
  }
}
console.log('total conflicts', conflicts); // should be zero using faker.unique()
console.log('total uniques generated', Object.keys(emails).length);

It should work for any faker method and will always ensure you will get a unique result. Has a few good options for configuration / safety to ensure CPU won't melt when pushing the boundaries finding unique results in set.

[Marak]

RE: #245

[Marak]

If everything looks good I'll be shipping faker.unique in next release. Let me know if you have any feedback.

[amackintosh]

Nice work. I love that technology direction in #245. If I have any feedback, it will be very specific. I think this will handle it, however. I look forward to next sampling, which might actually be soon since I'm working on a frontend right now.

[bonesoul]

same issue here, my tests randomly failing as faker.js can not guarantee uniqueness.

[Marak]

@bonesoul - Did you read this whole thread and or test the new functionality for uniqueness in master branch?

I'm not understanding your contribution to this conversation.

[bonesoul]

oh sorry didn't read it all, will check out the master.

[Marak]

Please do. It's very basic, but should be working.

Could use improvement.

[richchurcher]

@Marak thanks for this, being able to generate unique emails is really valuable for seeding and testing. Note that the current implementation in master is a little buggy due to the use of startTime in module scope. The elapsed time gets gradually longer in comparison to this time, so it will tend to throw. Easiest way to see this is in the REPL...

> const faker = require('faker')
undefined
> faker.unique(faker.internet.email)
'[email protected]'
> faker.unique(faker.internet.email)
'[email protected]'
> faker.unique(faker.internet.email)
found 2 unique entries before throwing error. 
retried: 0 
total time: 5044 ms
Error: exceeded maxTime for uniquness check. may not be able to generate any more unique values with current settings. try adjusting maxTime or maxRetries parameters for faker.unique()

Using the command history, I managed to get off two tries before it threw :smile: it wouldn't be so much of an issue, but I have a seed script that runs over several minutes and tanks on unique after five seconds.

Edit: simple fix, will PR.

[JasonCust]

@Marak Is this still live in 4.1?

> const faker = require('faker');
undefined
> faker.unique
undefined

[Marak]

@JasonCust

Try cloning the master branch from Github, it should be in there.

There is much improvement for uniqueness, especially around email APIs.

[JasonCust]

@Marak Works. Any idea of when it will be published to NPM?

[hansede]

This problem is happening because of this line:

switch (faker.random.number(2)) {

You are choosing from each email generation scheme with equal probability, but that means that you are not assigning equal probability to every possible outcome because some schemes generate far more possible outcomes than others. Given that there are ~3k first names in this repo, you are appending a number (00-99), and case 0 is selected one in three times, you have the following odds of getting a single outcome from case 0:

1 / (3009 first names * 100 numbers * 3 cases) or 1 in 902,700.

Given the birthday problem, collisions are very likely to happen in such a small set.

You could dramatically decrease the odds of a collision by removing cases 0 and 1 and always generating from case 2. From there you could further dramatically decrease collisions by appending faker.random.number(99999) Instead of faker.random.number(99).

Case 2 has the following odds by itself:

1 / (3009 first names * 476 last names * 2 comma or underscore * 100 numbers) or 1 in 286,456,800

If we increase the random number generation to have 5 digits, we can change the odds to 1 in 286 billion. Perhaps a good addition to the email function would be a parameter to specify the number of digits to append to the end of the address.

[Marak]

@hansede -

That actually makes a lot of sense. Thank you for taking the time to explain the issue.

I didn't personally write that switch case statement, but it's now obvious to me why we are seeing the high rate of collisions for this method. I'm going to rewrite the email generation API based on your suggestions. Should be easy.

To be clear, the addition of faker.unique is still a very good general solution across faker for ensuring uniqueness.

Fixing this newly identified issue specifically with email generation will be a simple and good fix for ensuring "more random" email addresses.

[pkuczynski]

@Marak, any plans to release a new version with faker.unique?

[pkuczynski]

@Marak ping

[fr3fou]

any news?

[rsaphala]

any updates on this?

[liquidvisual]

@Marak Ping

[malimccalla]

Would love to see this released soon

[Marak]

@tarikhamilton - I've deleted your comment and temporarily removed your ability to further contribute.

Next time please try to be more civil in your tone and comments.

[AbdealiJK]

Seems like we're having issues with uniqueness too - and came across this thread.
Is there a release in the works that we should be waiting for - or is the recommended approach to leverage faker.unique to use the github master branch directly ?

Maybe having a rc or beta version on npm registry would help users who want to test it out as this seems to be a very wanted feature :)

EDIT : Seems like I jumped the gun to post this comment as the discussion for the release is happening in https://github.com/Marak/faker.js/issues/692

[berniedurfee]

Bump 👍

[abettke]

Bumping here with enthusiastic interest

[Ben-Kincaid]

Also very interested in this.. currently using from master branch directly and .unique works great. But would love to see this in the main package, as @types/faker doesn't currently have types for .unique.

[elseify-r-medicalib]

Does anybody know how to use faker unique with faker.fake generator ?