I am trying to get the number of rows and number of columns after reading the file from csv. But I am unable to get the number of rows. Please suggest some solution.
test_data <- spark_read_csv(sc = sc, name = "test1",path ="C:/Users/user1/data/bk_test1.csv")
dim(test_data)
[1] NA 15
ncol(test_data)
[1] 15
nrow(test_data)
[1] NA
This is fixed in the development version of sparklyr
and will be going to CRAN soon. Install it with:
if (!require("devtools")) install.packages("devtools")
devtools::install_github("rstudio/sparklyr")
Thanks @kevinushey
Still doesn't work...
@solunsteve sdf_nrow()
Most helpful comment
@solunsteve
sdf_nrow()