Go: Duplicate UNSAFE_QUORUM entry in stellar-core temp config when reingesting range in Horizon + Captive Core Docker Setup

[lorenzobersano]

What version are you using?

Cloned this repo as in https://github.com/stellar/go/commit/17275e80a6821fd057ad3313b651a1ba07f9bf80

What did you do?

I started a Horizon + Captive Core setup using the start script in pubnet mode as found here https://github.com/stellar/go/tree/master/services/horizon/docker.
The base setup works fine, but once I get into the Horizon container the horizon db reingest range <from> <to> doesn't work .

What did you expect to see?

I expected to see the reingestion start correctly.

What did you see instead?

I'm getting this error:

INFO[2021-03-23T14:47:27.292Z] Ingestion system initial state                current_state="reingestHistoryRange(fromLedger=18587775, toLedger=34573119, force=false)" pid=307 service=ingest
INFO[2021-03-23T14:47:27.292Z] Preparing ledger backend to retrieve range    from=18587775 pid=307 service=ingest to=34573119
INFO[2021-03-23T14:47:28.063Z] default: Config from /tmp/captive-stellar-core996790614/stellar-core.conf  pid=307 service=ingest subservice=stellar-core
ERRO[2021-03-23T14:47:28.065Z] default: Got an exception: Failed to parse '/tmp/captive-stellar-core996790614/stellar-core.conf' :Key UNSAFE_QUORUM already present at line 13  pid=307 service=ingest subservice=stellar-core
ERRO[2021-03-23T14:47:28.065Z] default: Please report this bug along with this log file if this was not expected  pid=307 service=ingest subservice=stellar-core
ERRO[2021-03-23T14:47:28.075Z] Error in ingestion state machine              current_state="reingestHistoryRange(fromLedger=18587775, toLedger=34573119, force=false)" error="error preparing range: error starting prepare range: opening subprocess: error running stellar-core: error waiting for `stellar-core new-db` subprocess: error waiting for `stellar-core [new-db]` subprocess: exit status 1" next_state=stop pid=307 service=ingest
INFO[2021-03-23T14:47:28.076Z] Shut down                                     pid=307 service=ingest
2021/03/23 14:47:28 error preparing range: error starting prepare range: opening subprocess: error running stellar-core: error waiting for `stellar-core new-db` subprocess: error waiting for `stellar-core [new-db]` subprocess: exit status 1

[tamirms]

The append path is configured via an environment variable and it defines UNSAFE_QUORUM=true. But, when we generate the stellar core configuration during a prepare range command we also set UNSAFE_QUORUM=true in the generated configuration. So, the resulting configuration has the property set twice

https://github.com/stellar/go/blob/master/ingest/ledgerbackend/stellar_core_runner.go#L130-L135

    if r.mode == stellarCoreRunnerModeOffline {
        // In offline mode, there is no need to connect to peers
        lines = append(lines, "RUN_STANDALONE=true")
        // We don't need consensus when catching up
        lines = append(lines, "UNSAFE_QUORUM=true")
    }

When the mode is in offline mode we need to make sure we aren't setting properties that are already defined by the append path.

I think we can fix this issue when implementing https://github.com/stellar/go/issues/3442

[Shaptic]

After some investigation, it seems like this can't be a quick fix. I believe there's two conflicting things going on:

• On one hand, the Docker configuration needs to be valid for Captive Core to start up (meaning we need UNSAFE_QUORUM=true specified in the configuration).
• On the other, executing reingest and friends explicitly additionally set UNSAFE_QUORUM=true (as @tamirms described above) which collides with the previous bullet.

One way to resolve this in a hacky fashion would be something akin to:

    if r.mode == stellarCoreRunnerModeOffline {
        // In offline mode, there is no need to connect to peers
        lines = append(lines, "RUN_STANDALONE=true")
        // We don't need consensus when catching up
-       lines = append(lines, "UNSAFE_QUORUM=true")
+       if !strings.Contains(appendConfigContents, "UNSAFE_QUORUM") {
+           lines = append(lines, "UNSAFE_QUORUM=true")
+       }
    }

but this feels... wrong? I'd defer to #3442 getting done, instead (and more-importantly #3193).