Osu: Multiple issues on importing beatmaps from stable

[Iodid]

The issue #1454 happens again in 2017.11.20.0

However this time the beatmaps that originally caused it in #1454 were imported just fine and the issue occured on a later beatmap.

The symptons and error messages are the same as in #1454 so I guess that the cause may be not the exact same but pretty similar.

Also some new errors popped up during the importing process.

[peppy]

logs?

[Iodid]

Like before the logs consist of a loop of the same error messages.
Instead of the huge looping textwall, here's 2 iterations as example:

22.11.2017 02:42:20: ----------------------------------------------------------
22.11.2017 02:42:20: Database Log for Michael
22.11.2017 02:42:20: osu version unknown
22.11.2017 02:42:20: Running on Microsoft Windows NT 10.0.16299.0, 8 cores
22.11.2017 02:42:20: ----------------------------------------------------------
22.11.2017 02:42:20: Failed executing DbCommand (4ms) [Parameters=[@p0='?', @p1='?', @p2='?', @p3='?', @p4='?', @p5='?', @p6='?', @p7='?', @p8='?', @p9='?', @p10='?', @p11='?', @p12='?', @p13='?', @p14='?', @p15='?', @p16='?', @p17='?', @p18='?', @p19='?', @p20='?', @p21='?'], CommandType='Text', CommandTimeout='10']
22.11.2017 02:42:20: INSERT INTO "BeatmapInfo" ("AudioLeadIn", "BaseDifficultyID", "BeatDivisor", "BeatmapSetInfoID", "Countdown", "DistanceSpacing", "GridSize", "Hash", "Hidden", "LetterboxInBreaks", "MD5Hash", "MetadataID", "OnlineBeatmapID", "Path", "RulesetID", "SpecialStyle", "StackLeniency", "StarDifficulty", "StoredBookmarks", "TimelineZoom", "Version", "WidescreenStoryboard")
22.11.2017 02:42:20: VALUES (@p0, @p1, @p2, @p3, @p4, @p5, @p6, @p7, @p8, @p9, @p10, @p11, @p12, @p13, @p14, @p15, @p16, @p17, @p18, @p19, @p20, @p21);
22.11.2017 02:42:20: SELECT "ID"
22.11.2017 02:42:20: FROM "BeatmapInfo"
22.11.2017 02:42:20: WHERE changes() = 1 AND "ID" = last_insert_rowid();
22.11.2017 02:42:20: An exception occurred in the database while saving changes for context type 'osu.Game.Database.OsuDbContext'.
22.11.2017 02:42:20: Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while updating the entries. See the inner exception for details. ---> Microsoft.Data.Sqlite.SqliteException: SQLite Error 19: 'UNIQUE constraint failed: BeatmapInfo.OnlineBeatmapID'.
22.11.2017 02:42:20: bei Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
22.11.2017 02:42:20: bei Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.Execute(IRelationalConnection connection, DbCommandMethod executeMethod, IReadOnlyDictionary`2 parameterValues)
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.ExecuteReader(IRelationalConnection connection, IReadOnlyDictionary`2 parameterValues)
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.Execute(IRelationalConnection connection)
22.11.2017 02:42:20: --- Ende der internen Ausnahmestapelüberwachung ---
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.Execute(IRelationalConnection connection)
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.Execute(Tuple`2 parameters)
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(IReadOnlyList`1 entriesToSave)
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(Boolean acceptAllChangesOnSuccess)
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.DbContext.SaveChanges(Boolean acceptAllChangesOnSuccess)
22.11.2017 02:42:20: Failed executing DbCommand (0ms) [Parameters=[@p2='?', @p3='?', @p4='?', @p5='?', @p6='?'], CommandType='Text', CommandTimeout='10']
22.11.2017 02:42:20: INSERT INTO "BeatmapSetInfo" ("DeletePending", "Hash", "MetadataID", "OnlineBeatmapSetID", "Protected")
22.11.2017 02:42:20: VALUES (@p2, @p3, @p4, @p5, @p6);
22.11.2017 02:42:20: SELECT "ID"
22.11.2017 02:42:20: FROM "BeatmapSetInfo"
22.11.2017 02:42:20: WHERE changes() = 1 AND "ID" = last_insert_rowid();
22.11.2017 02:42:20: An exception occurred in the database while saving changes for context type 'osu.Game.Database.OsuDbContext'.
22.11.2017 02:42:20: Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while updating the entries. See the inner exception for details. ---> Microsoft.Data.Sqlite.SqliteException: SQLite Error 19: 'UNIQUE constraint failed: BeatmapSetInfo.OnlineBeatmapSetID'.
22.11.2017 02:42:20: bei Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
22.11.2017 02:42:20: bei Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.Execute(IRelationalConnection connection, DbCommandMethod executeMethod, IReadOnlyDictionary`2 parameterValues)
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.ExecuteReader(IRelationalConnection connection, IReadOnlyDictionary`2 parameterValues)
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.Execute(IRelationalConnection connection)
22.11.2017 02:42:20: --- Ende der internen Ausnahmestapelüberwachung ---
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.Execute(IRelationalConnection connection)
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.Execute(Tuple`2 parameters)
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(IReadOnlyList`1 entriesToSave)
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(Boolean acceptAllChangesOnSuccess)
22.11.2017 02:42:20: bei Microsoft.EntityFrameworkCore.DbContext.SaveChanges(Boolean acceptAllChangesOnSuccess)
22.11.2017 02:42:21: Failed executing DbCommand (0ms) [Parameters=[@p0='?', @p1='?'], CommandType='Text', CommandTimeout='10']
22.11.2017 02:42:21: INSERT INTO "FileInfo" ("Hash", "ReferenceCount")
22.11.2017 02:42:21: VALUES (@p0, @p1);
22.11.2017 02:42:21: SELECT "ID"
22.11.2017 02:42:21: FROM "FileInfo"
22.11.2017 02:42:21: WHERE changes() = 1 AND "ID" = last_insert_rowid();
22.11.2017 02:42:21: An exception occurred in the database while saving changes for context type 'osu.Game.Database.OsuDbContext'.
22.11.2017 02:42:21: Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while updating the entries. See the inner exception for details. ---> Microsoft.Data.Sqlite.SqliteException: SQLite Error 19: 'UNIQUE constraint failed: FileInfo.Hash'.
22.11.2017 02:42:21: bei Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
22.11.2017 02:42:21: bei Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
22.11.2017 02:42:21: bei Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.Execute(IRelationalConnection connection, DbCommandMethod executeMethod, IReadOnlyDictionary`2 parameterValues)
22.11.2017 02:42:21: bei Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.ExecuteReader(IRelationalConnection connection, IReadOnlyDictionary`2 parameterValues)
22.11.2017 02:42:21: bei Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.Execute(IRelationalConnection connection)
22.11.2017 02:42:21: --- Ende der internen Ausnahmestapelüberwachung ---

[Iodid]

I'm currently trying to locate the new beatmap that cuases the issue and record a video of it.
I've let it running over night due to a huge volume of beatmaps so I currently don't know which one caused it. If it was one of the later ones of my 9000 it may take a while until I reach that point again.

[Iodid]

I also happened upon a new error during import:
https://imgur.com/a/Bzitf

[Iodid]

Here's another new one:

[Aergwyn]

I did a change that shows the set that failed in that generic error message but that is not live (yet). That may help you identifying the responsible maps in the future.

[Iodid]

@peppy With the improved logging I've found one of the faulty beatmaps:
13023 Masayoshi Minoshima - Shinto Shrine (Instrumental)

[peppy]

could you please omit the screenshots and just include the text logs? they aren’t helpful

[Iodid]

Ah ok, sorry didn't know that.

It looks like my database.log didn't update with the newest error. After closing lazer the last entry in my database.log is from 3 days ago. The meta data indicates that the file was last edited 3 days ago as well.
The other logs updated though.

[peppy]

pretty sure you want runtime.log?

[Iodid]

Ah okay. The other error messages before were taken from the database.log. If that wasn't helpfull I apologize. Now I know better for future reference.

Here's the runtime log for this error:

[...]
25.11.2017 01:54:03: Focus contention triggered by NotificationOverlay (275,0) 320x768.
25.11.2017 01:54:30: Focus contention triggered by MusicController (0,40) 400x130.
25.11.2017 02:01:53: Focus contention triggered by NotificationOverlay (320,0) 320x768.
25.11.2017 02:01:56: Focus contention triggered by MainSettings (0,0) 0x768.
25.11.2017 02:02:00: Focus contention triggered by NotificationOverlay (305,0) 320x768.
25.11.2017 02:31:26: ERROR: Could not import beatmap set (13023 Masayoshi Minoshima - Shinto Shrine (Instrumental))
25.11.2017 02:31:26: System.IO.InvalidDataException: Unknown command type: �
25.11.2017 02:31:26: bei osu.Game.Beatmaps.Formats.OsuLegacyDecoder.handleEvents(Beatmap beatmap, String line, StoryboardSprite& storyboardSprite, CommandTimelineGroup& timelineGroup)
25.11.2017 02:31:26: bei osu.Game.Beatmaps.Formats.OsuLegacyDecoder.ParseFile(StreamReader stream, Beatmap beatmap)
25.11.2017 02:31:26: bei osu.Game.Beatmaps.Formats.BeatmapDecoder.ParseFile(StreamReader stream)
25.11.2017 02:31:26: bei osu.Game.Beatmaps.Formats.OsuLegacyDecoder.ParseFile(StreamReader stream)
25.11.2017 02:31:26: bei osu.Game.Beatmaps.Formats.OsuLegacyDecoder.Decode(StreamReader stream)
25.11.2017 02:31:26: bei osu.Game.Beatmaps.BeatmapManager.importToStorage(FileStore files, BeatmapStore beatmaps, ArchiveReader reader)
25.11.2017 02:31:26: bei osu.Game.Beatmaps.BeatmapManager.Import(ArchiveReader archiveReader)
25.11.2017 02:31:26: bei osu.Game.Beatmaps.BeatmapManager.Import(String[] paths)

[Iodid]

25.11.2017 03:57:06: ERROR: Could not import beatmap set (140301 Eluveitie - A Rose for Epona)
25.11.2017 03:57:06: System.IO.IOException: Unknown file format
25.11.2017 03:57:06: bei osu.Game.Beatmaps.Formats.BeatmapDecoder.GetDecoder(StreamReader stream)
25.11.2017 03:57:06: bei osu.Game.Beatmaps.BeatmapManager.importToStorage(FileStore files, BeatmapStore beatmaps, ArchiveReader reader)
25.11.2017 03:57:06: bei osu.Game.Beatmaps.BeatmapManager.Import(ArchiveReader archiveReader)
25.11.2017 03:57:06: bei osu.Game.Beatmaps.BeatmapManager.Import(String[] paths)

[FreezyLemon]

First error:
In the Normal diff's .osu file of this map, there's a Ñ character at line 46 which causes this error. There's definitely no case for this, but I'm also wondering whether the file reader is even set to deal with Unicode (since it shows a question mark and not the actual character, even inside VS)?

Second error:
The decoder says unknown file format because for some reason, the "BM's Hard" diff of http://osu.ppy.sh/d/140301 says ow computerosu file format v12 in its first line instead of just osu file format v12. Not sure how this wasn't/isn't an issue in the stable client..

[p20]

please remove us from this thread, we're not related to any of this. Thank you.

[smoogipoo]

You will need to unsubscribe from this issue thread @p20.

[FreezyLemon]

Tried to import all my beatmaps from stable today and encountered another issue:
This map's highest (7K) diff has no version number at all on the first line, and throws another error.

[FreezyLemon]

Had another weird one with "unknown file format" where the first character in the .osu file (encoding is not UTF8, but UTF8-BOM for some reason) is the "control" character (download, just the character).

[Aergwyn]

The map Love Is War can't be imported (even if it reports a success) because the files neither contain a BeatmapSetID nor a BeatmapID. Thus it's assumed as "nekodex - circles" and completes.

[peppy]

Same seems to be the case with https://osu.ppy.sh/s/8023

[peppy]

Can we open a separate issue to track this one? ie. .osu files without BeatmapID/BeatmapSetID present don't import at all.