Cartodb: Advanced import options

[danicarrion]

I'd like to start the discussion about whether it's a good idea to have an "advanced options" section in the editor, when importing.

Users could fine tune their import jobs there. Things like: specifically choosing their CSV separator, decimal separator, projection... Sometimes guessing simply doesn't work, and there is currently no way for users to try again with different options, they need to ask for support.

The first discussion would be to decide whether having this advanced options section or not. And then, if yes, discuss which options we could put there.

[danicarrion]

Another interesting option: disable geo-enrichment for twitter imports

[danicarrion]

I was reading GNIP's docs [1] again today and I found this sentence:

The “next” element works exactly the same for high-volume queries, allowing you to page through all of the results within a given minute.

If I get it right, it seems they're acknowledging the fact that they will only serve the last minute of high-volume queries. They're actually creating a feature from a bug.

I think this makes it even more necessary to have a way to disable geo-enrichment (and therefore avoid high-volume queries) in the editor.

[1] http://support.gnip.com/apis/search_api/api_reference.html#SearchRequests

cc @Kartones @javisantana

[jsanz]

QGIS almost covers all the typical annoyances when dealing with CSV importing:

• Skipping some initial lines that sometimes we found people using to store random information
• Defining the encoding of the file
• Defining non standard column names for lat/lon
• Using custom delimitiers instead of comma or tab

[andy-esch]

This would be wonderful! I'm trying to help a journalist get a CSV to sync from this "csv". The first two lines are metadata that don't have anything to do with the header on line three. This means that the dataset can't be sync'd with out of the box cartodb. We don't want to have to setup a proxy server to do intermediate processing.

[AbelVM]

:+1:

[AbelVM]

Skipping some initial lines that sometimes we found people using to store random information

[andrewbt]

https://xkcd.com/977/

(for when someone gives you a CSV of projected coordinates and you want to import it 😂 )

[danicarrion]

and, goes without saying, import API too ;-)

[makella]

Love this idea!

[nygeog]

Users in the state of Massuchessetts or other places w/ leading 0's in ZIP codes would like this. Of the new users I've talked with, many don't notice the checkbox for allowing field type guessing.

Pandas has great params for reading CSV and XLS that could serve for some inspiration.
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_excel.html

Skiprows for example would be great to allow.

[saleiva]

hi @nygeog for that purpose (and many others) we've redesigned the import modal. Will be implemented soon.

[alonsogarciapablo]

There has been no activity on this issue for more several months. We are closing it. If you think this still needs to be addressed please open a new issue.

[andy-esch]

@saleiva, this is being implemented?

[saleiva]

Not yet.

On Mon, Nov 21, 2016, 6:16 PM Andy Eschbacher [email protected]
wrote:

@saleiva https://github.com/saleiva, this is being implemented?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/CartoDB/cartodb/issues/4282#issuecomment-262003866,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAIENf1uEsC-yIzqczvM13AppmTbKloSks5rAdHmgaJpZM4FQ1_Z
.

[rafatower]

http://www.gdal.org/drv_csv.html