Mimesis: Randomness lost in multiprocessing

[chris-canipe]

Mimesis: 3.2.0
OS: macOS Mojave, v10.14.3
Python: 3.6.8

Situation

I'm creating a mass of fake data to test scaling — enough that I need to use multiprocessing for performance. However, doing so does not randomize the data among a set of workers.

The only solution I've found is to create a new mimesis object for each iteration, seeded with a count (see https://stackoverflow.com/a/29855961/1729586). This works, but performance takes a hit: in my case it takes ~7s to create 1,000 records when creating a seeded object for each iteration; if I only create the object once it takes ~1.5s.

Questions

1. Is there anything I can do improve performance?
2. Can a change be made to the internals that would work with multiprocessing?
3. Would creating one object and updating the seed be feasible?

Simplified Example

import numpy as np

from argparse import ArgumentParser
from mimesis import Generic
from multiprocessing import Pool

parser = ArgumentParser()
parser.add_argument('names', type=int, help='Number of names to generate')

def get_fake(i):
  # To see the issue, remove the "seed" argument below.
  return Generic('en', seed=np.random.RandomState(i))

def generate_name(i):
  fake = get_fake(i)
  return fake.person.full_name()

def main():
  args = parser.parse_args()
  with Pool() as p:
    for name in p.imap(generate_name, range(args.names)):
      print(name)

if __name__ == "__main__":
  main()

With the seed in place, the names appear randomly; if the seed is removed, the output looks something like this:

Tyson Dominguez
Tyson Dominguez
Tyson Dominguez
Tyson Dominguez
Tyson Dominguez
Tyson Dominguez
Tyson Dominguez
Tyson Dominguez
Tyson Dominguez
Tyson Dominguez
Tyson Dominguez
Tyson Dominguez
Vanesa O'brien
Vanesa O'brien
Vanesa O'brien
Vanesa O'brien
Vanesa O'brien
Vanesa O'brien
Vanesa O'brien
Vanesa O'brien

[lk-geimfari]

1. I'm not sure why do you create a new object each time. Also, if you need only data from Person use it directly, i.e person = Person('en', seed=SEED), because of Generic is slower than directly used providers.
2. Technically - yes, I guess, but I'm busy and I can't work on it right now.
3. Absolutely. You need to use the method .reseed(), just like this:

person.reseed(seed=NEW_SEED)

@sobolevn, @ceccoemi Any comments or recommendations?

[chris-canipe]

Thanks @lk-geimfari! I completely missed reseed in the docs and using it dropped my test from ~7s to ~2s. I'll let you know if I hit any other snags.

[lk-geimfari]

@epicyclist Glad, it helped you. I'll try to work on multiprocessing as soon as I will less busy (hopefully on this weekends).

Also, it's strange that generating 1000 takes 2 sec because I've tested it here and it worked much faster. Do you always need the same set of data?

[chris-canipe]

@lk-geimfari, my apologies for the confusion: the times are in reference to my actual, much more complex code — not the simplified example I posted.

[lk-geimfari]

@epicyclist I understand. Anyway, I hope I helped you.