Teslamate: Import from TeslaFi

[rikbrown]

It would be nice to have an automated import from TeslaFi (which I hope to migrate from). I think it can export all the data, but not sure on the format.

(This is something I may try to contribute once I’ve set up Teslamate, just creating this to track it or see if it is on your radar).

[bahree]

I have seen this request a few times and happy to take a crack at it and write a tool. The one thing though - I don't use TeslaFi - is there somewhere one can read up on its schema used and is there a export feature? Need to see what's the data structure is like.

Somewhere one of you can point me to or share this?

Cheers.

[tucopup]

TeslaFi allows export of data as a csv by month. Attached is a clip of an export.

Let me know if you need more/different.

TeslaFi82017 edit.zip

[bahree]

Thanks @tucopup ; let me try and make sense of this, and see how to map it to the schema of TeslaMate.

[bahree]

I started to look at this and wanted to create a documentation of the existing schema, so then to understand how to map TeslaFi's to this. I took a stab at this here. @adriankumpf , does this look directionally correct?

[Tigwin]

This is my most important issue to migrate off of TeslaFi. I don't want to lose all my historical data.

[mboerma]

Yes, this would be a great feature!

[adriankumpf]

I started to look at this and wanted to create a documentation of the existing schema, so then to understand how to map TeslaFi's to this. I took a stab at this here. @adriankumpf , does this look directionally correct?

Looks good to me, @bahree!

I have never used TeslaFi and therefore have no access to exported data. The CSV data from @tucopup seems to be quite complete though.

There are currently no plans to support an automated import from TeslaFi. Contributions are of course welcome though. So if anyone wants to contribute, that would be great!

[mel0en]

Is someone still working on this? I would like a one-time conversion as well.

[adriankumpf]

I may have found a way to import data but I have no idea if it'll work reliably. If someone agrees to provide me with _all_ their TeslaFi data (CSV export), that'd be great. At the moment it is difficult to test because I don't have enough data available. You can send it to my personal email (available on my profile) instead of posting it publicly.

[rikbrown]

@adriankumpf I'll share. Emailing.

[rikbrown]

Ah, this seems annoying. When I go to export from Teslafi (I'm going to Drives -> Search and Download -> (search for all) -> CSV) it doesn't include the addresses for locations I've labeled. e.g. "Home" is just "Home" not "". That won't be much good for import. Is there another place in TeslaFi I can export from anyone?

[tucopup]

TeslaFi: Settings; Account, bottom of the page in the last section, Download TeslaFi Data. Haven't looked at addresses, but will.

I sent Adrian my files as well.

[Tigwin]

salivates in anticipation

[olexs]

Huge thanks for this, and especially for including sample CSV files in test fixtures. I was able to export my data from a different logger I used before TeslaMate (https://github.com/lephisto/tesla-apiscraper), write a script to convert it to the TeslaFi CSV format, and import it all into TeslaMate. It took a half day of debugging, but it worked!

I will clean up the conversion script, and push step-by-step instructions to TeslaMate docs shortly.

[Tigwin]

Huge thanks for this, and especially for including sample CSV files in test fixtures. I was able to export my data from a different logger I used before TeslaMate (https://github.com/lephisto/tesla-apiscraper), write a script to convert it to the TeslaFi CSV format, and import it all into TeslaMate. It took a half day of debugging, but it worked!

How did you import it? I created the import/ folder, and added the entry in the data dir in the yml file, restarted, and I don't see the import screen.

[dani190]

Huge thanks for this, and especially for including sample CSV files in test fixtures. I was able to export my data from a different logger I used before TeslaMate (https://github.com/lephisto/tesla-apiscraper), write a script to convert it to the TeslaFi CSV format, and import it all into TeslaMate. It took a half day of debugging, but it worked!

How did you import it? I created the import/ folder, and added the entry in the data dir in the yml file, restarted, and I don't see the import screen.

ya same issue here... no import screen. Just shows the regular teslamate screen each time.

[olexs]

I had the same issue trying to get it to import when launching TeslaMate from source code locally (basically dev mode) - it wouldn't show the import screen, no matter how I set IMPORT_DIR and where I put the files.

Using the current docker-compose.yml to pull up a TeslaMate instance and placing the files into the "import" directory right next to it worked on first attempt.

I think it's important what the files are called - sample files in test fixtures and my output files from conversion follow the format TeslaFi<month><year>.csv, e.g. TeslaFi52019.csv (note the 5 without a leading zero). It seems that TeslaMate at least parses the filename to display the month in the import screen, even though I believe it's not important for the actual import process - but it might have to follow the format to be recognized correctly.

Also make sure you updated to v1.16 of course.

[dani190]

I had the same issue trying to get it to import when launching TeslaMate from source code locally (basically dev mode) - it wouldn't show the import screen, no matter how I set IMPORT_DIR and where I put the files.

Using the current docker-compose.yml to pull up a TeslaMate instance and placing the files into the "import" directory right next to it worked on first attempt.

I think it's important what the files are called - sample files in test fixtures and my output files from conversion follow the format TeslaFi<month><year>.csv, e.g. TeslaFi52019.csv (note the 5 without a leading zero). It seems that TeslaMate at least parses the filename to display the month in the import screen, even though I believe it's not important for the actual import process - but it might have to follow the format to be recognized correctly.

Also make sure you updated to v1.16 of course.

awesome - that worked.
My problem was understanding this piece of the docker-compose file

volumes: - ./import:/opt/app/import

I thought the files needed to go in /opt/app/import. Really what I needed to do was leave this alone, and when docker created my import folder next to my yml file, just add my csv files there.

Thanks!

[Tigwin]

My problem was understanding this piece of the docker-compose file

volumes: - ./import:/opt/app/import

I thought the files needed to go in /opt/app/import. Really what I needed to do was leave this alone, and when docker created my import folder next to my yml file, just add my csv files there.

Still doesn't work for me.

My teslamate/ folder is located at /home/ec2-user/teslamate.

docker-compose.yml contains:

volumes: - teslamate-db:/var/lib/postgresql/data - ./import:/opt/app/import

I ran:

docker-compose down docker-compose pull docker-compose up -d

Then after it started up there was a /home/ec2-user/teslamate/import/ folder (owned by root).

I then did a sudo cp to copy the csv files over. It now contains:

$ ls import TeslaFi102019.csv TeslaFi32019.csv TeslaFi62019.csv TeslaFi92019.csv TeslaFi112019.csv TeslaFi42019.csv TeslaFi72019.csv TeslaFi122019.csv TeslaFi52019.csv TeslaFi82019.csv

I then restarted teslamate again:

docker-compose down docker-compose pull docker-compose up -d

Teslamate and graphana interfaces look normal, it doesn't do any importing.

[MattTW]

Ditto - following directions - unable to get any "import" page to pop up. Using default directory location next to docker-compose.yml. Has TeslaFi files with correct names. I did ensure my docker compose file was up to date, most notably updating to postgres 12 w/backup-restore. After restart standard Teslamate dashboard only, no import screen.

If I remove the import directory, I do get the following error, so it is seeking out the directory at startup.

admin@DiskStation:~$ sudo docker-compose up

Creating network "admin_default" with the default driver
Creating admin_teslamate_1 ... error
Creating admin_database_1 ...
Creating admin_grafana_1 ...
Creating admin_database_1 ... done
Creating admin_grafana_1 ... done
Creating admin_mosquitto_1 ... done

ERROR: for teslamate Cannot start service teslamate: Bind mount failed: '/volume1/homes/admin/import' does not exists
ERROR: Encountered errors while bringing up the project.

When the import directory exists, it comes up without error, but navigating to the Teslmate UI home screen does not bring up any import screen:

admin@DiskStation:~$ sudo docker-compose up
admin_mosquitto_1 is up-to-date
Starting admin_teslamate_1 ...
admin_database_1 is up-to-date
Starting admin_teslamate_1 ... done
Attaching to admin_mosquitto_1, admin_database_1, admin_grafana_1, admin_teslamate_1
mosquitto_1 | WARNING: no logs are available with the 'db' log driver
database_1 | WARNING: no logs are available with the 'db' log driver
grafana_1 | WARNING: no logs are available with the 'db' log driver
teslamate_1 | WARNING: no logs are available with the 'db' log driver

[dani190]

Ditto - following directions - unable to get any "import" page to pop up. Using default directory location next to docker-compose.yml. Has TeslaFi files with correct names. I did ensure my docker compose file was up to date, most notably updating to postgres 12 w/backup-restore. After restart standard Teslamate dashboard only, no import screen.

If I remove the import directory, I do get the following error, so it is seeking out the directory at startup.

admin@DiskStation:~$ sudo docker-compose up
Creating network "admin_default" with the default driver
Creating admin_teslamate_1 ... error
Creating admin_database_1 ...
Creating admin_grafana_1 ...
Creating admin_database_1 ... done
Creating admin_grafana_1 ... done
Creating admin_mosquitto_1 ... done
ERROR: for teslamate Cannot start service teslamate: Bind mount failed: '/volume1/homes/admin/import' does not exists
ERROR: Encountered errors while bringing up the project.

When the import directory exists, it comes up without error, but navigating to the Teslmate UI home screen does not bring up any import screen:

admin@DiskStation:~$ sudo docker-compose up
admin_mosquitto_1 is up-to-date
Starting admin_teslamate_1 ...
admin_database_1 is up-to-date
Starting admin_teslamate_1 ... done
Attaching to admin_mosquitto_1, admin_database_1, admin_grafana_1, admin_teslamate_1
mosquitto_1 | WARNING: no logs are available with the 'db' log driver
database_1 | WARNING: no logs are available with the 'db' log driver
grafana_1 | WARNING: no logs are available with the 'db' log driver
teslamate_1 | WARNING: no logs are available with the 'db' log driver

Did you copy the docker-compose file exactly as is without editing the volume path?

[MattTW]

Did you copy the docker-compose file exactly as is without editing the volume path?

Yes

[adriankumpf]

One thing to keep in mind: it is not enough to just create the folder, the CSV files must already be placed in it before restarting.

If you are unsure where to create the folder, try using an absolute path instead of ./import.

@MattTW, on Synology the actual Docker logs can be found somewhere in the Docker UI afaik.

[adriankumpf]

Huge thanks for this, and especially for including sample CSV files in test fixtures. I was able to export my data from a different logger I used before TeslaMate (https://github.com/lephisto/tesla-apiscraper), write a script to convert it to the TeslaFi CSV format, and import it all into TeslaMate. It took a half day of debugging, but it worked!

I will clean up the conversion script, and push step-by-step instructions to TeslaMate docs shortly.

That's awesome! Looking forward to the PR.

[DineshCyanam]

@adriankumpf: I had almost 2 years of TeslaFi data to import and I had issues importing my data. There were 2 months (07/2018 and 11/2018) which were throwing errors while importing. Removing these 2 files from the import directory helped. Would you be interested in taking a look at the data? I can send you all my files.

Also, it looks like the import is a one time process? I tried to import one month at a time (import -> restart -> import...) and only the first-month data gets imported and the rest do not show up although I see a checkmark for the other months. In my import process, I had to do multiple DB wipes and restart the import process to determine which files were causing the import to fail and that is how I determined it was 07/2018 and 11/2018 files.

[dani190]

@adriankumpf: I had almost 2 years of TeslaFi data to import and I had issues importing my data. There were 2 months (07/2018 and 11/2018) which were throwing errors while importing. Removing these 2 files from the import directory helped. Would you be interested in taking a look at the data? I can send you all my files.

Also, it looks like the import is a one time process? I tried to import one month at a time (import -> restart -> import...) and only the first-month data gets imported and the rest do not show up although I see a checkmark for the other months. In my import process, I had to do multiple DB wipes and restart the import process to determine which files were causing the import to fail and that is how I determined it was 07/2018 and 11/2018 files.

Had this exact same problem. Mine was 11/2018 and 12/2018. Also noticed that the import process was a one time thing as I tried to do 2019/2020 and then 2018 separately.

[tucopup]

I imported 3 cars data, one at a time. I'm using Synology/Docker. There were two minor issues that occured. Otherwise everything went as expected.

First, the import would not start until I did chmod 777 on the csv files (permission complaint in the logs).

Second, the third car import aborted quickly; the logs contained the name of the second car. I assumed I mixed up csv files, but it turns out that in the first months csv for car #3, the first 40 lines referenced car #2. So I re-exported that month from Teslafi; the file was identical. So I omitted that month (actually only 1 day with one long drive) and everything went fine. Not a Teslamate problem, rather TeslaFi/Tesla.

This is really great!

[Tigwin]

I tried full path and chmod 777 on the import and csv files, still nothing happens. What url is supposed to have the new import screen?

[tucopup]

It opened when I went to [IP address]:4000, but my history says that it was [IP address]:4000/import

[Tigwin]

hmmm, I tried that. Gives me an error.

Internal Server Error

[adriankumpf]

@adriankumpf: I had almost 2 years of TeslaFi data to import and I had issues importing my data. There were 2 months (07/2018 and 11/2018) which were throwing errors while importing. Removing these 2 files from the import directory helped. Would you be interested in taking a look at the data? I can send you all my files.

Had this exact same problem. Mine was 11/2018 and 12/2018. Also noticed that the import process was a one time thing as I tried to do 2019/2020 and then 2018 separately.

Yeah, you can send me an email with the relevant files or open a new issue.

[dani190]

@adriankumpf: I had almost 2 years of TeslaFi data to import and I had issues importing my data. There were 2 months (07/2018 and 11/2018) which were throwing errors while importing. Removing these 2 files from the import directory helped. Would you be interested in taking a look at the data? I can send you all my files.

Had this exact same problem. Mine was 11/2018 and 12/2018. Also noticed that the import process was a one time thing as I tried to do 2019/2020 and then 2018 separately.

Yeah, you can send me an email with the relevant files or open a new issue.

I have sent you my Nov 2018 file which fails to import.

[MattTW]

my issue was a permissions issue, too. chmod 777 on the import dir and files fixed it. Thanks for the tips!

[SorenLan]

I tried to import TeslaFi files from Sep '19 up to Sep '20. I put the files in place and restarted TeslaMate. After that I get the Import page and I can see all my files. But after I pushed the Import button I got a red exclamation mark after each file and a error message. Anyone who can clarify what happened and what I can do to solve the problem?

I also tried to import only the file for Sep 2020 but when pushed the Import button nothing at all happens.

{{:function_clause, [ {CSV.Decoding.Preprocessing.Lines, :starts_sequence?, [ <<229, 100, 229, 114, 101, 110, 44, 78, 111, 110, 101, 44, 78, 111, 110, 101, 44, 44, 53, 89, 74, 51, 69, 55, 69, 65, 57, 75, 70, 51, 55, 48, 56, 53, 57, 44, 78, 111, 110, 101, 44, 52, ...>>, "l", false, 44, <<50, 48, 49, 57, 45, 48, 57, 45, 49, 48, 32, 49, 49, 58, 53, 52, 58, 53, 52, 34, 44, 84, 114, 117, 101, 44, 44, 49, 53, 55, 50, 51, 50, 49, 50, 50, 48, 44, ...>> ], []}, {CSV.Decoding.Preprocessing.Lines, :start_sequence, 3, [file: 'lib/csv/decoding/preprocessing/lines.ex', line: 85]}, {Stream, :do_transform_user, 6, [file: 'lib/stream.ex', line: 902]}, {Enumerable.Stream, :do_each, 4, [file: 'lib/stream.ex', line: 1609]}, {Stream, :do_transform, 5, [file: 'lib/stream.ex', line: 880]}, {Enumerable.Stream, :do_each, 4, [file: 'lib/stream.ex', line: 1609]}, {Stream, :do_transform, 5, [file: 'lib/stream.ex', line: 880]}, {Enumerable.Stream, :do_each, 4, [file: 'lib/stream.ex', line: 1609]} ]}, {GenServer, :call, [:api_370859, :get_vehicle, :infinity]}}