Pokeapi: MemoryError on building pokemon_v2_pokemonmove

[AlecGW]

It's happened the last 3 times I've tried to build this database and I can't manage to fix it. I'm inexperienced with this kind of stuff, so I have no idea what to do with this. The only thing that I've found that causes this otherwise is setting 'DEBUG' to true.
Here's the error log:

...
building pokemon_v2_pokemonmove
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "I:\PokeAPI\Newfolder\pokeapi-master\data\v2\build.py", line 2471, in build_all
    build_pokemons()
  File "I:\PokeAPI\Newfolder\pokeapi-master\data\v2\build.py", line 2209, in build_pokemons
    model.save()
  File "C:\Python27\lib\site-packages\django\db\models\base.py", line 591, in save
    force_update=force_update, update_fields=update_fields)
  File "C:\Python27\lib\site-packages\django\db\models\base.py", line 616, in save_base
    with transaction.commit_on_success_unless_managed(using=using, savepoint=False):
  File "C:\Python27\lib\site-packages\django\db\transaction.py", line 300, in __enter__
    connection._start_transaction_under_autocommit()
  File "C:\Python27\lib\site-packages\django\db\backends\sqlite3\base.py", line 466, in _start_transaction_under_autocommit
    self.cursor().execute("BEGIN")
  File "C:\Python27\lib\site-packages\django\db\backends\utils.py", line 88, in execute
    'time': "%.3f" % duration,
MemoryError
>>>

I get the feeling it's part of an installation I did badly, but I'm not sure and I've re-installed things to try and getting it to work, but I'm not sure...

[mjhayes412]

I'm having the same issue. Perhaps by serializing the csv files into a json file, this could be resolved using django loaddata. I'll tinker with it and let you know if it works for me.

[OrangeCrush]

@mjhayes412 @AlecGW I'm curious- what are the specs of your build machine?? Also - what platform are you building on? (Windows, Linux, ...etc)

Just curious as I have built the project hundreds of times (Linux + Docker) without seeing this error.

[AlecGW]

@OrangeCrush Oh yeah I forgot to mention my build in the original post. I'm running Windows (10) so that's probably the problem there. I was considering setting up a Linux VM, but my computer has been acting up lately, so I don't want to try anything like that at the moment.
As for my specific specs, I'm not sure what to mention other than maybe that I have an Intel i7-4790 CPU @ 3.60GHz and 8GB of RAM.

[mjhayes412]

Hopefully I don't embarrass myself too much with my inexperience. I'm running it on a Linux VM via Digital Ocean. Currently with 1 GB Memory / 20 GB Disk / NYC3 - Ubuntu Django on 14.04

[OrangeCrush]

I'm going to set up a small (1GB ram) test environment and run the build overnight to see if I am able to reproduce the error as well.

[mjhayes412]

This is absolutely fascinating to me. I'm not familiar with this technology but that's why I'm tinkering around.

For the PokemonMove table alone, I tested the time it took to complete in my environment and it was about 20 minutes.

I read that the model.save() call is really expensive and having it inside the iteration is really killing the time. So I added an array and pushed each object in it, then I used bulk_create() to save all them at once. This cut me down to 14 minutes.

Best case scenario would be to load in a fixture of the data using django-admin loaddata. Tested that and it got me down to 3 minutes.

Not sure what the solution should be here but either way I would love to contribute so I can get more experience with all this.

[OrangeCrush]

I was able to run the build and force the machine out of memory..
However the error I see is a little different, so I don't think that's the issue you 2 are having.

building pokemon_v2_pokemonspecies
building pokemon_v2_pokemonspeciesname
building pokemon_v2_pokemonspeciesdescription
building pokemon_v2_pokemonspeciesflavortext
building pokemon_v2_pokemon
building pokemon_v2_pokemonsprites
building pokemon_v2_pokemonability
building pokemon_v2_pokemondexnumber
building pokemon_v2_pokemonegggroup
building pokemon_v2_pokemonevolution
building pokemon_v2_pokemonform
building pokemon_v2_pokemonformsprites
building pokemon_v2_pokemonformname
building pokemon_v2_pokemonformgeneration
building pokemon_v2_pokemongameindex
building pokemon_v2_pokemonhabitatname
building pokemon_v2_pokemonitem
building pokemon_v2_pokemonmove
Segmentation fault

@mjhayes412 I think it would be fantastic if we could get the DB build time down.. if you want to put a branch up or have something on your fork I'd be more than happy to test!

[phalt]

Anything that reduces the time to make it easier to load data, would make me very happy.

The original implementation was a hack from the old days of Django and not intended for performance :)

[nimaje]

I started to use functions in the build script to let objects faster fall out of scope, but never finished this (I only removed top-level code and writen the function style for _build_languages). I wanted a method to save all models in one db query for process_csv but I had not found something.
We could use something like this http://stackoverflow.com/a/13290055, but the function is named django.db.transaction.atomic now.

The problem with bulk_create is that you could only use it for one model class but sometimes we create two different models out of one csv row, like in build_damage_classes (https://github.com/phalt/pokeapi/blob/master/data/v2/build.py#L300-L312) (this is why data_to_model is a generator).

[mjhayes412]

Hm, for that special case, could we use 2 arrays and push each object respectfully then have 2 bulk_create calls?

I think I want to go the route of using loaddata though. That seemed the fastest to me. I'm under the assumption that future game data will be added in the same manner (by csv) so I'd like to plan for that. I only tried it once but I had an issue after executing the django-admin command in build.py. Couldn't access the next table so I suppose there's a closure issue.

By the way, this is awesome. I'm new to GitHub but I'm loving this interaction so far.

[nimaje]

The problem I see with using loaddata is to get the data into the needed format (This also needs time). I don't think we would want to use one of the needed formats for the data.

Could somebody please try this:

python manage.py shell
from data.v2.build import build_all
from django.db import transaction
transaction.atomic(build_all)()

This should only create one big db query.
This _could_ reduce time and _could_ need the same amount of memory.
(I do not have posgresql installed (and don't want to install it) and with sqlite it needs much more time)

There is at least one more case where we create two models.

now I have searched for them and found 9 more:
1 https://github.com/phalt/pokeapi/blob/master/data/v2/build.py#L665-L695
2 https://github.com/phalt/pokeapi/blob/master/data/v2/build.py#L772-L783
3 https://github.com/phalt/pokeapi/blob/master/data/v2/build.py#L916-L928
4 https://github.com/phalt/pokeapi/blob/master/data/v2/build.py#L1052-L1064
5 https://github.com/phalt/pokeapi/blob/master/data/v2/build.py#L1087-L1099
6 https://github.com/phalt/pokeapi/blob/master/data/v2/build.py#L1251-L1263
7 https://github.com/phalt/pokeapi/blob/master/data/v2/build.py#L1697-L1709
8 https://github.com/phalt/pokeapi/blob/master/data/v2/build.py#L1969-L2000
9 https://github.com/phalt/pokeapi/blob/master/data/v2/build.py#L2082-L2120

[mjhayes412]

Could we use the dumpdata command on an established build and save it as json file on the repo? Then instead of using data.v2.build_all, include something in the Makefile to build from the json file? Sure, new data will arise and come in csv format, but should just one person do the dirty work instead of the entire project?

Idk, just thinking out loud.

[phalt]

@mjhayes412 Welcome to GitHub! Glad you're enjoying this so far :)

Feel free to suggest ideas, it's a great way of collaborating.

A few things that would be nice to have that I've experienced with running this in production:

1) When we get data updates, we upload _all the content_ even though 90% of it already exists, if we could write a script that knows what's been loaded and what hasn't, that would reduce load times.

2) We could supply a "dev data" version that has the bare minimum to be functional in development and only has 20% of the data, which would also reduce load times.

We could take advantage of our test suite to dump data stuff when changes are made to the code and then shipped to the master branch, instead of having to do it manually.

[mjhayes412]

I think those are great ideas. So here's what I've found out:

1) Instead of uploading _all the content_ every time, I want to try and use the get_or_create function. It's a get() except it will create the object if it doesn't exist. Plus, it creates and saves while it's still on the database side so that cuts out the extra model.save() step.

This method could be hazardous if not done correctly. If we edit a record in the database, it could cause this build function to insert duplicates. Which do you think is better:

a) insert new record or do nothing to existing record
b) insert new record or update existing record to match csv

2) Fixtures are the way to go for initial data, in my opinion. When I tested, it only took 4 minutes to load the entire database as opposed to the 29 minutes with build_all(). We could supply a full database load and the partial load you mentioned for dev. I say just the original 151 for the dev load?

So, I found out having the model.save() inside an iteration is super costly but I think another big factor is mapping the foreign keys. The serialized json objects map them just on ID but in the build_all() function, we're having to model.get() _every_ reference for the whole object. That's got to be a killer cost.

By the way, what's a good Python editor/IDE? I'm using Brackets and it's not a big fan of Python.

[phalt]

1) get_or_create() is brilliant, great idea. I never touch or modify live data so it's perfectly okay to use that.

2) Fixtures is great, we just need a way to transform all the raw data into fixtures every time we change the data.

Atom is a great editor for Python, with a bunch of plug ins

[mjhayes412]

Not having so much luck with the get_or_create(), to be honest. I'm going with the approach of instantiating a queryset of the entire table before going into the for loop. That way it'll cut so much of the back-and-forth with the db. I'm going to keep at it. I do see value in it still when new data needs to be added in bulk via veekun's csv files.

Out of curiosity, is there any particular reason the ManyToManyField option isn't ever used in the model?

[mjhayes412]

@nimaje whoa, I went full circle and now realize you were way ahead of me on this one. I think I'm caught up (or closer to being) and I'm going to look over your branch data_outsourcing. This is such an awesome way to get hands-in something new, holy crap!

[nimaje]

@mjhayes412 if you look at the commits you will see that I only provide a way to change the data location (by changing the load function). I forgot that I pushed that to github.
@phalt should I make a PR or do we not want this functionality?

@mjhayes412 because of your question about a editor/IDE for python:
I use vim as editor. It has syntax highlighting for alot of formats build in and there are lots of plugins for vim. You could read Unix as IDE

By the way pypy is great (and needs less memoy than cpython arcording to their site), could we try to be and stay compatible to pypy (I think we are but I don't know)