Dataframes.jl: Port dependencies to Missing and CategoricalArray

[nalimilan]

This issue lists packages which depend on DataFrames, with links to PRs which port them to the new Missing and CategoricalArray based DataFrames. Please add new PRs as they are opened to track progress, and check boxes once they have been merged. Some packages may not need any adjustments and could be moved to a separate section.

Packages with open PRs (tick the box once PR is merged):

• [x] CSV: https://github.com/JuliaData/CSV.jl/pull/95
• [x] CovarianceMatrices: https://github.com/gragusa/CovarianceMatrices.jl/commit/0d12f70c8cd82251257918014a479f042c3864c2
• [x] DataFramesMeta: https://github.com/JuliaStats/DataFramesMeta.jl/pull/79
• [x] ExcelFiles
• [x] ExcelReaders
• [x] Feather: https://github.com/JuliaStats/Feather.jl/pull/48
• [x] FixedEffectModels: https://github.com/matthieugomez/FixedEffectModels.jl/pull/20
• [x] FreqTables: https://github.com/nalimilan/FreqTables.jl/pull/16
• [x] Gadfly: https://github.com/GiovineItalia/Gadfly.jl/pull/1088 https://github.com/GiovineItalia/Gadfly.jl/pull/1090
• [x] GeoStats
• [x] GeoStatsDevTools
• [x] IterableTables: https://github.com/davidanthoff/IterableTables.jl/pull/60, https://github.com/davidanthoff/IterableTables.jl/pull/62
• [x] LogParser
• [x] Microeconometrics: https://github.com/lbittarello/Microeconometrics.jl/commit/ef496df7527e9060e8626bb415362908e6c71b12
• [x] Mimi
• [x] MixedModels: https://github.com/dmbates/MixedModels.jl/pull/103
• [x] MySQL: https://github.com/JuliaDB/MySQL.jl/pull/91 https://github.com/JuliaDB/MySQL.jl/pull/92
• [x] ODBC: https://github.com/JuliaDB/ODBC.jl/pull/173
• [x] PlotlyJS: https://github.com/sglyon/PlotlyJS.jl/pull/145
• [x] Quandl: milktrader/Quandl.jl#111
• [x] Query: https://github.com/davidanthoff/Query.jl/pull/155
• [x] QueryOperators: https://github.com/davidanthoff/QueryOperators.jl/pull/2
• [x] RCall
• [x] RData: https://github.com/JuliaStats/RData.jl/pull/28
• [x] RDatasets: https://github.com/johnmyleswhite/RDatasets.jl/pull/48 (package works with 0.11 already with deprecation warnings)
• [x] ReadStat
• [x] ScikitLearn: https://github.com/cstjean/ScikitLearn.jl/commit/9555a438b2c7b3d90577958b213a9c462abc7dd9
• [x] SQLite: https://github.com/JuliaDB/SQLite.jl/pull/131
• [x] StatFiles: https://github.com/davidanthoff/StatFiles.jl/commit/a171896d6df02e7ac0deb03b4ab1218f0efee18b
• [x] StatsModels: https://github.com/JuliaStats/StatsModels.jl/pull/30
• [x] Taro: https://github.com/aviks/Taro.jl/pull/38
• [x] UAParser

Packages which do not need any changes:

• [x] StatPlots
• [x] Twitter

Packages which need new PRs:

• [ ] BeaData
• [ ] Bedgraph
• [ ] BedgraphFiles
• [ ] Benchmark
• [ ] BioMedQuery
• [ ] BlsData
• [ ] Bootstrap
• [ ] Brim
• [ ] COMTRADE
• [ ] Celeste
• [ ] CrossfilterCharts
• [ ] DBFTables
• [ ] DSGE
• [ ] DataCubes
• [ ] Deldir
• [ ] DimensionalityReduction
• [ ] Diversity
• [ ] EEG
• [ ] EasyPhys
• [ ] EconDatasets
• [ ] EconModels
• [ ] EventHistory
• [ ] ExperimentalAnalysis
• [ ] FaSTLMM
• [ ] FinancialMarkets
• [ ] FredData
• [ ] GeoIP
• [ ] Gillespie
• [ ] GoogleCharts
• [ ] Graft
• [ ] GraphGLRM
• [ ] Hive
• [ ] InteractiveFixedEffectModels
• [ ] JWAS
• [ ] JuliaFEM
• [ ] LazyQuery
• [ ] LifeTable
• [ ] LinguisticData
• [ ] LowRankModels
• [ ] MLDataUtils
• [ ] MachineLearning
• [ ] Mads
• [ ] mPulseAPI
• [ ] MultidimensionalTables
• [ ] NLOptControl
• [ ] NLreg
• [ ] OdsIO
• [ ] OpenGene
• [ ] OpenSecrets
• [ ] OptiMimi
• [ ] PGFPlots
• [ ] PhyloNetworks
• [ ] PrettyPlots
• [ ] ProjectTemplate
• [ ] PySide
• [ ] Q
• [ ] ROCAnalysis
• [ ] Resampling
• [ ] Rif
• [ ] Robotlib
• [ ] RobustStats
• [ ] SloanDigitalSkySurvey
• [ ] Sparrow
• [ ] SpatialEcology
• [ ] SpeedDate
• [ ] StackedNets
• [ ] TermWin
• [ ] TextAnalysis
• [ ] TimeData
• [ ] TimeSeriesIO
• [ ] ValueOrientedRiskManagementInsurance
• [ ] WorldBankData
• [ ] XSim
• [ ] ZipCode

Abandoned packages:

• [x] kNN

Packages that already did not load on Julia 0.6:

• [x] Augur
• [x] BIGUQ
• [x] BayesNets
• [x] BayesianDataFusion

[ararslan]

Oh man.

[rofinn]

I can probably take a look at Bootstrap and Resampling next week. I've been wanting to transfer Bootstrap over to JuliaStats anyways (assuming the author is open to that).

[quinnj]

I think we should get an initial tag out of the port; trying to update all this code and have travis runs isn't possible unless we're able to test against current DataFrames. We don't need to announce anything yet and make sure we have really tight bounds on everything, but I think we need to tag sooner rather than later.

[nalimilan]

I think we should get an initial tag out of the port; trying to update all this code and have travis runs isn't possible unless we're able to test against current DataFrames.

That's possible by adding an explicit call to Pkg.checkout("DataFrames") in .travis.yml (see JuliaStats/RData.jl#28). Not pretty, but as a temporary way of checking the PR it works.

I think we should have the discussion about when to tag the port somewhere else, maybe in https://github.com/JuliaData/DataFrames.jl/pull/1209 since it has started there?

[dmbates]

I think it would be useful to write up some guidelines on how to convert a dependent package to the new version of DataFrames, CategoricalArrays, etc. To even be able to install the master version of DataFrames I ended up starting from a clean package directory by setting the environment variable JULIA_PKGDIR. I don't know if this is a good approach or not but I do know that it is easy to find yourself in a "twisty maze of passages" (so who is old enough to know where that phrase originated?).

[dmbates]

Carrying on with this "clean slate" approach (I can move this to another issue if desired). After setting the environment variable JULIA_PKGDIR to, say, ~/.juliaDF11 and starting julia, run

Pkg.init()
Pkg.add("DataFrames")
Pkg.checkout("DataFrames")
versioninfo(true)

the output on packages is

Package Directory: /home/bates/.juliaDF11/v0.6
1 required packages:
 - DataFrames                    0.10.1+            master
13 additional packages:
 - BinDeps                       0.6.0
 - CategoricalArrays             0.2.0
 - Compat                        0.31.0
 - DataStreams                   0.2.0
 - DataStructures                0.7.1
 - Nulls                         0.0.6
 - Reexport                      0.0.3
 - SHA                           0.5.1
 - SortingAlgorithms             0.1.1
 - SpecialFunctions              0.3.3
 - StatsBase                     0.19.0
 - URIParser                     0.2.0
 - WeakRefStrings                0.3.1

Do any of the additional packages need to have a Pkg.checkout before I continue?

[quinnj]

That looks right to me.

[piever]

StatPlots is compatible with DataFrames via the IterableTables package, and given that IterableTables has already updated, it doesn't need a PR.

[jmkuhn]

Quandl: milktrader/Quandl.jl#111 has been merged into master.

[juliohm]

Please mark GeoStatsDevToosl.jl as fixed. I am just waiting for the new tag to be merged on METADATA.jl and then I will tag a new version of GeoStats.jl. It is currently broken with the update.

[juliohm]

GeoStats.jl fixed as well: https://github.com/JuliaLang/METADATA.jl/pull/12225

[Datseris]

If anyone is seeing this: If you can't install DataFrames 11, it may be because of DataTables. For me that was what was holding my upgrade! After I removed DataTables Dataframes updated!

[randyzwitch]

UAParser.jl updated for missings, just needs to be merged on METADATA

[randyzwitch]

LogParser.jl update is at METADATA, just needs to be merged

[randyzwitch]

Per https://github.com/johnmyleswhite/kNN.jl/issues/20#issuecomment-174555460, kNN.jl is abandoned

[randyzwitch]

Twitter.jl technically already works, but updates coming with https://github.com/randyzwitch/Twitter.jl/pull/17 to make DataFrames 0.11 a requirement

[davidanthoff]

ReadStat.jl no longer depends on DataFrames, so it can be removed from the list.

IterableTables.jl, QueryOperators.jl and Query.jl should also all work with DataFrames v0.11 at this point, so no further work needed there.

[randyzwitch]

PR in for Taro.jl https://github.com/aviks/Taro.jl/pull/38

[matthieugomez]

I am trying to migrate but it's more complicated than what I expected.
Could someone give me a hand with this particular function?
https://discourse.julialang.org/t/tanslate-code-from-pooleddataarray-to-categoricalarray/8646

[Ken-B]

Could you uncheck the Gadfly box as the PR remains unmerged?
It took me a while to find out that it was the one holding back my update. Thanks.

[nalimilan]

It's not checked, an IIRC it wasn't a few hours ago either, was it?

[Ken-B]

I would have sworn it wasn't, but perhaps I'm mistaken, in which case sorry for the noise.

[davidanthoff]

Alright, ExcelFiles.jl and ExcelReaders.jl can (finally) be marked as resolved on this list. ExcelReaders.jl no longer depends on DataFrames.jl, and ExceFiles.jl no longer on DataTables.jl, and both packages should work fine with the new v0.11 now.

[davidanthoff]

Mimi.jl can also be marked as solved.

[nalimilan]

Closing since packages which haven't been ported don't work on Julia 1.0, so they have more serious problems.