Framework: Collection::toArray: Error when parsing timestamp(6) with zero microseconds in postgres

[ndtuan412]

• Laravel Version: 5.7.19
• PHP Version: PHP Version 7.2.11-4+ubuntu18.04.1+deb.sury.org+1
• Database Driver & Version:
  
  
  
  • PostgreSQL(libpq) Version: 10.5 (Ubuntu 10.5-0ubuntu0.18.04)
  
  
  • PostgreSQL(libpq): PostgreSQL 10.5 (Ubuntu 10.5-0ubuntu0.18.04) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0, 64-bit
  
  
• Homestead: v7.19.2

Description:

(I know there are many similar issues out there #6673, #5612, #5623, #5762)
I encountered this issue several times, tried several ways to bypass with luck, but not today.
So currently, the framework expects that the timestamp got from result set always exists the microsecond part even if it is all zero. But actually it doesn't.

With some settings in model, it just works as expected most of the time:

class BaseModel extends Model {
    protected $dateFormat = 'Y-m-d H:i:s.u O';

    public function getCreatedAtAttribute($value) {
        // try parsing the $value with several format
    }
}

But the method $collection->toArray() doesn't respect the mutator, so no chances for me to bypass this issue.

Like every one else, I think we can do some way:

1. Make method toArray (and maybe toJson) respect the mutator.
2. OR use a second config $fallbackDateFormat and we can try each config like:

class BaseModel extends Model {
    protected $dateFormat = 'Y-m-d H:i:s.u O';
    protected $fallbackDateFormat = 'Y-m-d H:i:s O';

    public function getCreatedAtAttribute($value) {
        // try parsing the $value with several date format
    }
}

Illuminate/Database/Eloquent/Concerns/HasAttributes.php

protected function asDateTime($value)
{
    //...

    try {
        return Carbon::createFromFormat(
            str_replace('.v', '.u', $this->getDateFormat()), $value
        );
    } catch (\Exception $e) {
        return Carbon::createFromFormat(
            str_replace('.v', '.u', $this->getFallbackDateFormat()), $value
        );
    }
}

Steps To Reproduce:

1.

create table test (
    created_at timestamp(6) with time zone
);
insert into test values (now());
insert into test values (date_trunc('second', now()));

2.

class BaseModel extends Model {
    protected $dateFormat = 'Y-m-d H:i:s.u O';

    public function getCreatedAtAttribute($value) {
        // try parsing the $value with several format
    }
}

class Test extends BaseModel {
    protected $table = 'test';
}

$models = Test::all();  // OK
$models->toArray(); // Error



md5-6c3b6c563b4b05f91a7f10d8db6a55bb



[2019-01-15 10:06:22] local.ERROR: Unexpected data found.
Data missing {...,"exception":"[object] (InvalidArgumentException(code: 0): Unexpected data found.
Data missing at /var/www/vendor/nesbot/carbon/src/Carbon/Carbon.php:910)
[stacktrace]
#0 /var/www/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php(797): Carbon\\Carbon::createFromFormat('Y-m-d H:i:s.u O', '2019-01-15 10:0...')
#1 /var/www/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php(129): Illuminate\\Database\\Eloquent\\Model->asDateTime('2019-01-15 10:0...')
#2 /var/www/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php(91): Illuminate\\Database\\Eloquent\\Model->addDateAttributesToArray(Array)
#3 /var/www/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php(1076): Illuminate\\Database\\Eloquent\\Model->attributesToArray()
#4 /var/www/vendor/laravel/framework/src/Illuminate/Support/Collection.php(1828): Illuminate\\Database\\Eloquent\\Model->toArray()
#5 [internal function]: Illuminate\\Support\\Collection->Illuminate\\Support\\{closure}(Object(App\\Models\\Test))
#6 /var/www/vendor/laravel/framework/src/Illuminate/Support/Collection.php(1829): array_map(Object(Closure), Array)
#7 /var/www/app/Http/Controllers/TestController.php(114): Illuminate\\Support\\Collection->toArray()
...
"} 

[driesvints]

I'll need some time to deep-dive into this. Any help is much appreciated in the mean time.

[rafaelbernard]

@ndtuan412 , this also affects me. I solved this problem locally with a trait, but it would be great if the framework could deal with this, maybe something like cast property does for other fields that are not created/updated.

My trait (maybe can help you for now):

<?php

namespace App;


use Carbon\Carbon;
use DateTimeInterface;

trait FixTimestampsDateFormat
{
    protected function asDateTimeCustomized($value)
    {
        // If this value is already a Carbon instance, we shall just return it as is.
        // This prevents us having to re-instantiate a Carbon instance when we know
        // it already is one, which wouldn't be fulfilled by the DateTime check.
        if ($value instanceof Carbon) {
            return $value;
        }

        // If the value is already a DateTime instance, we will just skip the rest of
        // these checks since they will be a waste of time, and hinder performance
        // when checking the field. We will just return the DateTime right away.
        if ($value instanceof DateTimeInterface) {
            return new Carbon(
                $value->format('Y-m-d H:i:s.u'), $value->getTimezone()
            );
        }

        // If this value is an integer, we will assume it is a UNIX timestamp's value
        // and format a Carbon object from this timestamp. This allows flexibility
        // when defining your date fields as they might be UNIX timestamps here.
        if (is_numeric($value)) {
            return Carbon::createFromTimestamp($value);
        }

        // If the value is in simply year, month, day format, we will instantiate the
        // Carbon instances from that format. Again, this provides for simple date
        // fields on the database, while still supporting Carbonized conversion.
        if ($this->isStandardDateFormat($value)) {
            return Carbon::createFromFormat('Y-m-d', $value)->startOfDay();
        }

        // Finally, we will just assume this date is in the format used by default on
        // the database connection and use that format to create the Carbon object
        // that is returned back out to the developers after we convert it here.

        $formatTimestampTZ = 'Y-m-d H:i:s.u T';
        $formatTimestamp   = 'Y-m-d H:i:s T';

        $changedDateFormat = $this->getDateFormat() === $formatTimestampTZ ? $formatTimestamp : $formatTimestampTZ;

        try {
            return Carbon::createFromFormat(
                str_replace('.v', '.u', $this->getDateFormat()), $value
            );
        } catch (\InvalidArgumentException $e) {

            return Carbon::createFromFormat(
                str_replace('.v', '.u', $changedDateFormat), $value
            );
        }
    }

    protected function addDateAttributesToArray(array $attributes)
    {
        foreach ($this->getDates() as $key) {
            if (!isset($attributes[$key])) {
                continue;
            }

            if (!in_array($key, [static::CREATED_AT, static::UPDATED_AT])) {
                $attributes[$key] = $this->serializeDate(
                    $this->asDateTime($attributes[$key])
                );
            }

            $attributes[$key] = $this->serializeDate(
                $this->asDateTimeCustomized($attributes[$key])
            );
        }

        return $attributes;
    }

}

[ndtuan412]

@rafaelbernard I intend to fork this repo to apply a temporarily fix inside framework directly. But your code inspired me to try some hack.
And this is my hack now (tested and worked like a charm haha). This will effect to all date time field.

BaseModel.php

use Illuminate\Database\Eloquent\Model;

class BaseModel extends Model {
    protected $dateFormat = '...';

    use DateTimeFix;
}

DateTimeFix.php

trait DateTimeFix {
    protected function asDateTime($value) {
        try {
            return parent::asDateTime($value);
        } catch (\Exception $e) {
            // try parsing the received value from database with some well-known date format,
            // or just the one you concern
            return DateUtility::tryParsedDateFromFormat($value);
        }
    }
}

Also I totally agree with you that the framework should handle this.
Or may we leave a placeholder as a last chance to parse date time. @driesvints What do you think about this? Any side-effects?

[staudenmeir]

Quick note on the mutator: toArray() does respect the mutator, but it gets called after the date is formatted with asDateTime(). If that fails, the mutator is never reached.

This is different from Model::getAttribute() where asDateTime() is not called when a mutator exists.

[driesvints]

In Laravel 7, a new default for date serialization will be used: https://github.com/laravel/framework/pull/30715

Also https://github.com/laravel/framework/pull/30628 recently got merged on master so this might be fixed now. Can you please try out dev-master to see if this is solved?

[ndtuan412]

In Laravel 7, a new default for date serialization will be used: #30715

Also #30628 recently got merged on master so this might be fixed now. Can you please try out dev-master to see if this is solved?

Tested on 7.0-dev (at commit 5d50d30c) and it's solved. Thank for your work.