Moor: _DefaultValueSerializer issue

[salim-lachdhaf]

The _DefaultValueSerializer class (in data_class.dart) did not cover all primitive variable type
for example if you call int x = fromJson<int>('10); it crashes
is it possible to add all primitive variable type like int, double, float, bool

class _DefaultValueSerializer extends ValueSerializer {
  const _DefaultValueSerializer();

   @override
  T fromJson<T>(dynamic json) {
    if (T == DateTime) {
      if (json == null) {
        return null;
      } else {
        return DateTime.fromMillisecondsSinceEpoch(json as int) as T;
      }
    }

    if (T == double && json is int) {
      return json.toDouble() as T;
    }

    // blobs are encoded as a regular json array, so we manually convert that to
    // a Uint8List
    if (T == Uint8List && json is! Uint8List) {
      final asList = (json as List).cast<int>();
      return Uint8List.fromList(asList) as T;
    }

    return json as T;
  }

  ...
}

[simolus3]

I'm not sure if it should handle those cases at all. The List<int> special logic is because that's how Uint8Lists are encoded in json, we obviously loose their type. It's reasonable that 0 can be read as a double, but I'd say that '0' should not. In general I prefer to keep the implementation as simple as possible, and special cases for other primitive types shouldn't be necessary.

If you need support to read strings as ints or other primitive types, you can always override the default serializer with moorRuntimeOptions.defaultSerializer. I'll close this issue as I don't think that should be the default. If you disagree and have a common scenario that would require this, let me know.

[salim-lachdhaf]

Sadly there a lot of common scenario that would require this, howover as you know we have IntColumn() ..., and if we try create an object fromJson() and one of column has column type as IntColumn and in json we have that value as String this will cause a problem.

[simolus3]

I see that, but why would the json have a string when an int is expected? Other json generators like json_serializable or built_value wouldn't accept that either. It sounds like an error on the server-side or whoever is generating the json if it represents integers as strings.

[salim-lachdhaf]

No, it's not a server problem, because in my case , i have a csv (all fields are strings) files that represents tables data, so i created a generic function<T extensds DataClass> that return T, by generate a json and call T.fromJson() to get my dataClass entity. I think at least the pluging should take in consideration int and bool

[simolus3]

Ok, but you'll agree that this is a very specific use-case, right? Since you know that you only get strings, it might make sense to map manually based on the column type:

extension MapFromCsvExtension<TableDsl extends Table, D extends DataClass>
    on TableInfo<TableDsl, D> {
  D mapFromCsvRow(List<String> values) {
    final typedValues = <String, dynamic>{};

    for (var i = 0; i < values.length; i++) {
      final rawValue = values[i];
      final column = $columns[i];

      dynamic typedValue;

      if (column is TextColumn) {
        typedValue = rawValue;
      } else if (column is BoolColumn) {
        typedValue = rawValue == 'true';
      } else if (column is IntColumn) {
        typedValue = int.parse(rawValue);
      } else if (column is DateTimeColumn) {
        typedValue =
            DateTime.fromMillisecondsSinceEpoch(int.parse(rawValue) * 1000);
      } else if (column is RealColumn) {
        typedValue = double.parse(rawValue);
      } else if (column is BlobColumn) {
        // don't know how you represent blobs in CSV
      }

      typedValue[column.$name] = typedValue;
    }

    return map(typedValues);
  }
}

This would allow you to use table.mapFromCsvRow from your database. I think the json converter isn't really a suitable tool when reading untyped values like CSV.

[salim-lachdhaf]

Very nice, seems a good solution , thank you for both your fast replys and your clever solution.
And i just advice to add int and bool as i say in future it will be useful, it will take just four lines :)

[salim-lachdhaf]

is it possible to make that extension returns a UpdateCompanion instead of DataClass ?
i tried my best but i didn't find any solution.

extension MapFromCsvExtension<TableDsl extends Table, D extends DataClass>
    on TableInfo<TableDsl, D> {
  Future<List<D>> loadFromCsv(File csvFile) async {
    if (csvFile == null || !await csvFile.exists()) return List();

    //Get all LINES from CSV Table
    List<String> csvRows = await csvFile.readAsLines(encoding: latin1);
    //Get all COLUMNS NAME from CSV Table
    List<String> csvColumns = CsvParser.decompose(csvRows[0], ';');

    if (csvColumns.isNullOrEmpty || $columns.isNullOrEmpty) return List();

    //each row  is a table data
    List<D> result = List();
    for (int row = 1; row < csvRows.length; row++) {
      //get row values
      List<String> values = CsvParser.decompose(csvRows[row], ';');

      //generate a data class for each row of data
      final typedValues = <String, dynamic>{};
      for (var i = 0; i < $columns.length; i++) {
        final localColumn = $columns[i];

        //escape the non existent columns
        if (!csvColumns.contains(localColumn.$name)) continue;

        //get cell value
        final cellValue = values[csvColumns.indexOf(localColumn.$name)];

        dynamic typedValue;

        if (localColumn is TextColumn) {
          typedValue = cellValue;
        } else if (localColumn is BoolColumn) {
          typedValue = cellValue == 'true';
        } else if (localColumn is IntColumn) {
          typedValue = int.parse(cellValue);
        } else if (localColumn is DateTimeColumn) {
          typedValue =
              DateTime.fromMillisecondsSinceEpoch(int.parse(cellValue) * 1000);
        } else if (localColumn is RealColumn) {
          typedValue = double.parse(cellValue);
        } else if (localColumn is BlobColumn) {
          // represent blobs in CSV
        }

        typedValues[localColumn.$name] = typedValue;
      }

      result.add(map(typedValues));
    }
    return result ?? List();
  }
}

[simolus3]

In moor 3.1 (which will hopefully release later today, fingers crossed), you can call toCompanion(false) to convert a data class into a companion.