Julia: slow DateTime parsing

[tanmaykm]

julia> const datefmt = Dates.DateFormat("yyyy-mm-dd HH:MM:SS.sss");

julia> length(datestr)
6668829

julia> @time d = DateTime(datestr, datefmt);
135.635752 seconds (634.26 M allocations: 21.842 GB, 9.04% gc time)

Comparing with strptime:

julia> function strpdatetime(a)
           l = length(a)
           d = Array(DateTime, l)
           for i in 1:l
               x = Libc.strptime("%Y-%m-%d %H:%M:%S", a[i])
               d[i] = DateTime(x.year+1900, x.month, x.mday, x.hour, x.min, x.sec)
           end
           d
       end
strpdatetime (generic function with 1 method)

julia> @time z = strpdatetime(datestr);
  1.092760 seconds (6.67 M allocations: 458.192 MB)

[omus]

Interesting. Can you give some details on what datestr looks like? Does it contain more than one date in the string? Does it matter where the date(s) are located in the string?

[tanmaykm]

datestr is a vector of ASCII strings. Sorry, I should have made that clear in the code snippet.

[ViralBShah]

I am guessing with Faker.jl, it should be possible to create a dataset that shows this performance issue.

[omus]

I believe this demonstrates the issue:

julia> const df = Dates.DateFormat("yyyy-mm-dd HH:MM:SS.sss");

julia> @time datestr = [Dates.format(dt, df) for dt in DateTime(2000,1,1,0):Dates.Millisecond(1):DateTime(2000,1,1,1)];
 34.667846 seconds (272.33 M allocations: 11.455 GB, 51.79% gc time)

julia> @time d = DateTime(datestr, df);
 75.166492 seconds (357.75 M allocations: 11.190 GB, 7.41% gc time)

[simonbyrne]

I think the date parsing code needs a major rethink. Looking at the code, there are lots of cases of non-concrete fields, type instabilities, etc., such that I think it ultimately needs a fairly big overhaul.

The one advantage we have is that users are only ever going to define a handful of date formats, and this will typically be done at a high-level scope (i.e. not within loops). As a result, this seems like a good candidate for code generation. There are a couple of ways to do this, but a simple approach would be to make DateFormat a parametric type, which takes a symbol containing the format specifier:

immutable DateFormat{S}
end

and create a string macro:

DateFormat"yyyy-mm-dd HH:MM:SS.sss"

which would expand to something like:

begin
    s = symbol("yyyy-mm-dd HH:MM:SS.sss")

    function parse(::DateFormat{s}, s::AbstractString)
        # generated parsing code
    end
    function print(io::IO, ::DateFormat{s}, dt::DateTime)
        # generated printing code
    end
    DateFormat{s}()
end

[simonbyrne]

cc: @quinnj

[nalimilan]

@simonbyrne Are you sure it is fine to define functions from inside macros like this? Wouldn't the user get "overwritten method" warnings if they create the same format twice in the same scope (in particular, twice at the global scope in different places)?

(I'm asking because a very similar strategy could be useful in StringEncodings.jl to avoid the need for generated functions.)

[JeffBezanson]

For now it seems like a good idea to have fast hand-written code for the most common formats, e.g. ISO 8601.

[quinnj]

I have some hand-written code for ISO 8601 in CSV.jl right now; I can clean that up as a fast-path for Base.

[femtotrader]

I'm also facing this slow DateTime parsing issue.

Here is a "minimal" code sample to show it with my use case:

I try to read 1 month of tick data of AUD/USD

Sample data can be found here
https://drive.google.com/file/d/0B8iUtWjZOTqla3ZZTC1FS0pkZXc/view?usp=sharing

AUDUSD-2014-01.zip is 11M and contains AUDUSD-2014-01.csv which is 85M
which is not so big!
It's in fact 1.947.106 rows like

AUD/USD,20140101 21:55:34.404,0.88796,0.88922

symbol,datetime,bid,ask

const S_FMT_DT = "yyyymmdd HH:MM:SS.sss"
const FMT_DT = Dates.DateFormat(S_FMT_DT)

const N = 100000

function parse_datetime(s::AbstractString)
    DateTime(
        parse(Int64, s[1:4]), # yyyy
        parse(Int64, s[5:6]), # mm
        parse(Int64, s[7:8]), # dd
        parse(Int64, s[10:11]), # HH
        parse(Int64, s[13:14]), # MM
        parse(Int64, s[16:17]), # SS
        parse(Int64, s[19:end])  # sss
    )
end

function parse_datetime_libc(s::AbstractString)
    x = Libc.strptime("%Y%m%d %H:%M:%S.%f", s)
    DateTime(x.year+1900, x.month+1, x.mday, x.hour, x.min, x.sec)  # ToFix! millisecond
end

function doit()
    t0 = time()
    fname = "AUDUSD-2014-01.csv"
    f = open(fname);
    for (i, ln) in enumerate(eachline(f))
        ln = ln[1:end-1]
        symb, s_dt, bid, ask = split(ln, ",")
        #dt = DateTime(s_dt, S_FMT_DT)  # parse DateTime (method 1a)
        #dt = DateTime(s_dt, FMT_DT)  # parse DateTime (method 1b)
        dt = parse_datetime(s_dt) # parse DateTime (method 2)
        #dt = parse_datetime_libc(s_dt) # parse Datetime (method 3)
        bid = parse(Float64, bid)
        ask = parse(Float64, ask)
        spread = ask - bid
        spread_pips = spread * 100000
        if i % N == 0
            #println("$i $ln")
            println("$i $symb $ask $bid $spread_pips")
            println(i / (time() - t0))
        end
    end
end

@time doit()

On my computer (MacBook Air Mid 2011 i5 1.7Ghz)

Method 1a: Processing @ 5492.941564379832 rows / s
Method 1b: Processing @ 20085.140649483576 rows / s (x4 from 1a)
Method 2: Processing @ 195435.98718016074 rows / s (x35 from 1a)
Method 3: Processing @ 76758.95271199026 (x14 from 1a but with millisecond issue)

Processing one month of tick data longs 9.3 seconds using (raw) method 2.
method 1b is 71.4s long.
I haven't been patient enough to wait end of task using method 1a.

[femtotrader]

I'm also testing @tanmaykm idea with Libc.strptime

But caution - month are 0 indexed in TmStruct

so

d[i] = DateTime(x.year+1900, x.month, x.mday, x.hour, x.min, x.sec)

should be replaced by

d[i] = DateTime(x.year+1900, x.month+1, x.mday, x.hour, x.min, x.sec)

Unfortunately I don't know how to (if we can) get milliseconds from TmStruct

[tanmaykm]

Libc.strptime doesn't parse milliseconds.
I used this C routine subsequently: https://github.com/tanmaykm/Chrono.jl/blob/master/deps/src/iso8601.c.
And @simonbyrne had also demonstrated Julia code that was as fast.

[simonbyrne]

Here is some code I wrote that is about 100x faster than the current implementation for parsing It RFC3339-formatted dates:
https://github.com/simonbyrne/DateParsing.jl/blob/master/src/tryparse.jl

We could have this be the default, and then also provide more customised parsers.