Light-client: Run the scenarios every night

[christianbrb]

Description

We want to run the scenarios each night as it is currently done for the Raiden Python client.

## Acceptance criteria
-

Tasks

• [x] Meeting with @deepbrook
• [x] Makethe SP player changes -> check first if this is needed for the nightly run
  
  
  
  • [x] https://github.com/raiden-network/scenario-player/issues/593
  
  
  • [x] https://github.com/raiden-network/scenario-player/issues/594
  
  
  • [x] https://github.com/raiden-network/scenario-player/issues/595
  
  
• [x] Provide a shell script to download the latest cli bundle (https://github.com/raiden-network/scenarioplayer.ci.raiden.network/issues/1)
  
  
  
  • Went for a docker image now
  
  
• [ ] Setup a server to run the nightlies (https://github.com/raiden-network/scenarioplayer.ci.raiden.network/issues/1)

[christianbrb]

Meeting with @deepbrook

1. He needs a shell script similiar to
   https://github.com/raiden-network/ansible/blob/master/playbooks/scenario-player/files/run-nightlies.sh
   a) Format Logging
   b) RC-Channel
   c) Everything needed to run the Light Client
   d) If the scenarios are different, provide a separate folder (we should avoid that)
2. He needs dedicated wallets

To align with @ulope @Dominik1999

1. Some changes André has proposed -> Create issues within the SP player repo
2. Change the scenario player to accept a Raiden Version, which can be a script

[andrevmatos]

https://github.com/raiden-network/scenario-player/issues/593
https://github.com/raiden-network/scenario-player/issues/594
https://github.com/raiden-network/scenario-player/issues/595

[christianbrb]

We would need to work with @ulope in future.

[christianbrb]

@ulope is currently working on setting up the nightly for the python client. @weilbith would be available to assist and collaborate.

[weilbith]

I contacted Ulo yesterday and got the link to the resource repository today. I'll take a look into that soon to get familiar with the setup.

[christianbrb]

@palango to provide the logs for the out of memory failure @andrevmatos to support :)

[palango]

> docker build -f Dockerfile.light_client .
Sending build context to Docker daemon  70.14kB
Step 1/9 : FROM node:14
 ---> f47907840247
Step 2/9 : RUN npm install -g pnpm
 ---> Using cache
 ---> a07774549edf
Step 3/9 : RUN git clone --recurse-submodules https://github.com/raiden-network/light-client.git
 ---> Using cache
 ---> 10b5272bab6a
Step 4/9 : RUN cd light-client
 ---> Using cache
 ---> e68f7b1565c9
Step 5/9 : RUN pnpm install --prefer-frozen-lockfile
 ---> Using cache
 ---> 530c0e521aba
Step 6/9 : RUN pnpm run build --filter raiden-ts
 ---> Running in 374bc3176d6e

<--- Last few GCs --->

[7:0x4d9dc20]   207612 ms: Mark-sweep 1949.0 (1980.7) -> 1942.2 (1981.5) MB, 3215.1 / 0.0 ms  (average mu = 0.287, current mu = 0.183) task scavenge might not succeed
[7:0x4d9dc20]   211345 ms: Mark-sweep 1949.5 (1981.5) -> 1942.8 (1982.2) MB, 3593.9 / 0.0 ms  (average mu = 0.176, current mu = 0.037) task scavenge might not succeed
[7:0x4d9dc20]   215137 ms: Mark-sweep 1950.2 (1982.2) -> 1943.3 (1982.0) MB, 3647.4 / 0.0 ms  (average mu = 0.111, current mu = 0.038) task scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa03010 node::Abort() [node]
 2: 0x94e471 node::FatalError(char const*, char const*) [node]
 3: 0xb76fbe v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb77337 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd31bd5  [node]
 6: 0xd3275f  [node]
 7: 0xd407eb v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 8: 0xd9ffcc v8::internal::ScavengeJob::Task::RunInternal() [node]
 9: 0xc6a1fb non-virtual thunk to v8::internal::CancelableTask::Run() [node]
10: 0xa711e4 node::PerIsolatePlatformData::RunForegroundTask(std::unique_ptr<v8::Task, std::default_delete<v8::Task> >) [node]
11: 0xa73049 node::PerIsolatePlatformData::FlushForegroundTasksInternal() [node]
12: 0x13792d6  [node]
13: 0x138bdb5  [node]
14: 0x1379c08 uv_run [node]
15: 0xa43444 node::NodeMainInstance::Run() [node]
16: 0x9d12d5 node::Start(int, char**) [node]
17: 0x7f6ef80c42e1 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
18: 0x96971c  [node]
Aborted
The command '/bin/sh -c pnpm run build --filter raiden-ts' returned a non-zero code: 134

[andrevmatos]

These are some weird errors.. I remember seeing something like that only when @eorituz was trying to transpile the SDK on the rPi.. our typesystem compilation is quite memory-intensive. It shouldn't occur on desktop/servers. Is there any memory constraint on the container or the host?

[andrevmatos]

There seems to be some issues and ways to increase node's heap limit:
https://github.com/angular/angular-cli/issues/5618#issuecomment-431310735
https://stackoverflow.com/questions/30252905/nodejs-decrease-v8-garbage-collector-memory-usage
Please, check if it works with some of these.

[palango]

I tried increasing the node memory to 6GB, but it still fails. I'm starting to wonder how this runs in CircleCI.

Step 6/9 : RUN NODE_OPTIONS=--max-old-space-size=6096 pnpm run build --filter raiden-ts
 ---> Running in 683bfd0d5bed

<--- Last few GCs --->

[6:0x621ec00]   458932 ms: Mark-sweep 5880.6 (6120.5) -> 5870.5 (6120.5) MB, 10549.2 / 0.1 ms  (average mu = 0.236, current mu = 0.030) task scavenge might not succeed
[6:0x621ec00]   470163 ms: Mark-sweep 5884.3 (6120.5) -> 5872.6 (6120.0) MB, 11071.0 / 0.0 ms  (average mu = 0.136, current mu = 0.014) task scavenge might not succeed
[6:0x621ec00]   481354 ms: Mark-sweep 5886.5 (6120.0) -> 5874.7 (6119.2) MB, 11041.3 / 0.0 ms  (average mu = 0.078, current mu = 0.013) task scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa03010 node::Abort() [node]
 2: 0x94e471 node::FatalError(char const*, char const*) [node]
 3: 0xb76fbe v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb77337 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd31bd5  [node]
 6: 0xd3275f  [node]
 7: 0xd407eb v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 8: 0xd9ffcc v8::internal::ScavengeJob::Task::RunInternal() [node]
 9: 0xc6a1fb non-virtual thunk to v8::internal::CancelableTask::Run() [node]
10: 0xa711e4 node::PerIsolatePlatformData::RunForegroundTask(std::unique_ptr<v8::Task, std::default_delete<v8::Task> >) [node]
11: 0xa73049 node::PerIsolatePlatformData::FlushForegroundTasksInternal() [node]
12: 0x13792d6  [node]
13: 0x138bdb5  [node]
14: 0x1379c08 uv_run [node]
15: 0xa43444 node::NodeMainInstance::Run() [node]
16: 0x9d12d5 node::Start(int, char**) [node]
17: 0x7fb54ec702e1 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
18: 0x96971c  [node]
Aborted (core dumped)
The command '/bin/sh -c NODE_OPTIONS=--max-old-space-size=6096 pnpm run build --filter raiden-ts' returned a non-zero code: 134

[palango]

@christianbrb Only the cron job is missing. I hope to fix that today and will then close it

[christianbrb]

Cool, thanks for the update :)

[weilbith]

@palango
Just because we talked about it this morning and you said eventually Systemd is a better fit than Cronjob, but you don't know how. Here is a tiny help for you:

scenarioplayer.service

[Unit]
Description=...
Requires= # Eventually you need a network connection or so?

[Service]
Type=oneshot
ExecStart= # Run the script here
ExecStart= # You can run more command. There are also Pre and Post executives...

scenarioplayer.timer

[Unit]
Description=Run ScenarioPlayer every night

[Timer]
OnCalendar=*-*-* 00:00:00

[Install]
WantedBy=timers.target  # I guess the default target is fine or do you need to orchestrate this with other services?

Store those files somewhere. Important is that they share the same base name!

systemctl link ./scenarioplayer.service
systemctl enable ./scenarioplayer.timer

If you are no root and act as a normal user you can use --user for the Systemd user environment (also saver with a different cgroup). If you want to start the timer now you could either define --now while enabling or just start the oneshot service itself with systemctl start scenarioplayer. To get a simple status check and see the last output of the script use systemctl status scenarioplayer. Note that without any suffix it assumes .service. If you want to act on the timer explicitly sate scenarioplayer.timer. Then you can use journalctl --unit scenarioplayer to see the logs. Other helpful flags are --boot (logs since last boot) and --follow. But now you have tons of functionality to read/filter/parse the logs including auto rotation etc. (man journalctl is your friend).

[palango]

Mostly works, scheduling is tracked in #2241