Go: Check performance bottlenecks/possible out of memory issues in the new ingestion system

[bartekn]

Given we have some extra time this sprint I think we should do some performance testing before we release the beta.

There are a couple of places in the code that consume RAM memory where it can be easily avoided. One of such issues is: https://github.com/stellar/go/issues/2128.

The result of this issue could be a statement like:

The new system can ingest x million ledger entry changes (distinct) in y seconds using z MB of RAM using c5.large AWS instance.

EDIT: where y < 5 so we know what's the max of ledger entry changes we can process and not be behind stellar-core more than 1 ledger at all times.

This way we know exactly what are the limits and what we are prepared for.

[ire-and-curses]

That statement would be incredibly helpful to have.

[bartekn]

To follow up on this: we're going to use c5.xlarge instead of c5.large because we actually used the former while batch testing. So let's find the max entries we can process with 8GB of RAM (and 4GB if there's time).

Would be great if it was a Horizon command so we can use this in the future.

[ire-and-curses]

Before we close this issue with the associated PR #2239, can we produce the statement Bartek posed?

The new system can ingest x million ledger entry changes (distinct) in y seconds using z MB of RAM using c5.large AWS instance.

[tamirms]

I created a kubernetes deployment to run the stress test from https://github.com/stellar/go/pull/2239 :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: horizon-stress-test
  namespace: sandbox
  labels:
    app: horizon-stress-test
spec:
  replicas: 1  # Number of pods that should be running
  selector:
    matchLabels:
      app: horizon-stress-test
  template:
    metadata:
      labels:
        app: horizon-stress-test
    spec:
      containers:
      - name: horizon
        image: stellar/horizon:1f24f3a54e727d57ddbc365717a9f06e03be1af9
        resources:
          requests:
            cpu: 2
            memory: 4Gi
          limits:
            cpu: 2
            memory: 4Gi
        env:
        - name: DATABASE_URL
          value: "postgres://postgres@localhost:5432/horizon?sslmode=disable"
        - name: STELLAR_CORE_DATABASE_URL
          value: "postgres://postgres@localhost:5432/horizon?sslmode=disable"
        - name: NETWORK_PASSPHRASE
          value: "Public Global Stellar Network ; September 2015"
        - name: STELLAR_CORE_URL
          value: "http://localhost:11626"
        - name: HISTORY_ARCHIVE_URLS
          value: "http://history.stellar.org/prd/core-live/core_live_001"
        args:
        - "--apply-migrations"
        - "expingest"
        - "stress-test"
      - name: postgres
        image: postgres:9.6-alpine
        resources:
          requests:
            cpu: 2
            memory: 4Gi
          limits:
            cpu: 2
            memory: 4Gi
        env:
        - name: POSTGRES_DB
          value: "horizon"

I configured the resource limits to match an AWS c5.large instance (4gb ram and 2 CPUs). The horizon instance was actually able to process all 4 million changes from the stress test ledger. However, after running the change processors, we create a new db ledger reader so we can run the transaction processors. It was at this phase that the kubernetes pod was killed because of the memory limits.

Here are some graphs showing CPU and memory consumption

[tamirms]

until we implement https://github.com/stellar/go/issues/2128 , I think it makes sense to change the ingestion code to reuse the same db ledger reader for both change processors and transaction processors. If we implemented such a change we would only read the ledger once and we probably would have been able to completely ingest the stress test ledger on an a c5.large instance

[tamirms]

I just finished running the same kubernetes deployment using resource limits which match an AWS c5.xlarge instance (8gb of RAM and 4 cpus). The stress test completed successfully (both the change processors and the transaction processors). This was expected given the results from the c5.large test. Here are the graphs showing CPU and memory usage:

[ire-and-curses]

To summarise: the graphs show that

The new system can ingest 4 million ledger entry changes (distinct) in ~20 minutes using 4.7GB of RAM using a c5.xlarge AWS instance.

See the YAML config above for the Postgres resources allocated for this test.