Docker-gitlab: default examples both docker-compose.yml and docker run commands don't work.

Created on 15 Jul 2018  Â·  7Comments  Â·  Source: sameersbn/docker-gitlab

Docker run command used

docker run --name gitlab -d \
    --link gitlab-postgresql:postgresql --link gitlab-redis:redisio \
    --publish 10022:22 --publish 10080:80 \
    --env 'GITLAB_PORT=10080' --env 'GITLAB_SSH_PORT=10022' \
    --env 'GITLAB_SECRETS_DB_KEY_BASE=V4VMktNNXWP9pp7Fmhb3b3jWKLgMcqXjJJgmqgdKHXntmCwCHgfsKjrTFgHkFp9F' \
    --env 'GITLAB_SECRETS_SECRET_KEY_BASE=LVvLTCTdqd43rq7zw3Lr3gPtjRpqqf7PdtCCNthz79grFPJ4VqqkhR9pwgmbv9zn' \
    --env 'GITLAB_SECRETS_OTP_KEY_BASE=pN3sCHgRh44JXcdHKzXHXrbXxPRx9dkCJHtmCVNvhgvr3XzNcXxspLc9Pmr7r4WL' \
    --volume $(pwd)/gitlab/gitlab:/home/git/data \
    sameersbn/gitlab:11.0.3```

## Output from docker logs -f

docker logs -f gitlab
Loading /etc/docker-gitlab/runtime/env-defaults
Initializing logdir...
Initializing datadir...
Installing configuration templates...
Configuring gitlab...
Configuring gitlab::database
Configuring gitlab::redis
Configuring gitlab::secrets...
Configuring gitlab::sidekiq...
Configuring gitlab::gitaly...
Configuring gitlab::monitoring...
Configuring gitlab::gitlab-workhorse...
Configuring gitlab::unicorn...
Configuring gitlab::timezone...
Configuring gitlab::rack_attack...
Configuring gitlab::ci...
Configuring gitlab::artifacts...
Configuring gitlab::lfs...
Configuring gitlab::uploads...
Configuring gitlab::mattermost...
Configuring gitlab::project_features...
Configuring gitlab::oauth...
Configuring gitlab::ldap...
Configuring gitlab::cron_jobs...
Configuring gitlab::backups...
Configuring gitlab::registry...
Configuring gitlab::pages...
Configuring gitlab-shell...
Configuring nginx...
Configuring nginx::gitlab...
2018-07-15 20:44:09,667 CRIT Supervisor running as root (no user in config file)
2018-07-15 20:44:09,667 WARN Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing
2018-07-15 20:44:09,667 WARN Included extra file "/etc/supervisor/conf.d/gitaly.conf" during parsing
2018-07-15 20:44:09,668 WARN Included extra file "/etc/supervisor/conf.d/gitlab-workhorse.conf" during parsing
2018-07-15 20:44:09,668 WARN Included extra file "/etc/supervisor/conf.d/mail_room.conf" during parsing
2018-07-15 20:44:09,668 WARN Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
2018-07-15 20:44:09,668 WARN Included extra file "/etc/supervisor/conf.d/sidekiq.conf" during parsing
2018-07-15 20:44:09,668 WARN Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing
2018-07-15 20:44:09,668 WARN Included extra file "/etc/supervisor/conf.d/unicorn.conf" during parsing
2018-07-15 20:44:09,676 INFO RPC interface 'supervisor' initialized
2018-07-15 20:44:09,676 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2018-07-15 20:44:09,677 INFO supervisord started with pid 563
Setting up GitLab for firstrun. Please be patient, this could take a while...
2018-07-15 20:44:10,679 INFO spawned: 'gitaly' with pid 579
2018-07-15 20:44:10,682 INFO spawned: 'sidekiq' with pid 580
2018-07-15 20:44:10,685 INFO spawned: 'unicorn' with pid 584
2018-07-15 20:44:10,688 INFO spawned: 'gitlab-workhorse' with pid 587
2018-07-15 20:44:10,690 INFO spawned: 'cron' with pid 588
2018-07-15 20:44:10,698 INFO spawned: 'nginx' with pid 589
2018-07-15 20:44:10,700 INFO spawned: 'sshd' with pid 590
2018-07-15 20:44:11,885 INFO success: gitaly entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-15 20:44:11,886 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-15 20:44:11,886 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-15 20:44:11,886 INFO success: gitlab-workhorse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-15 20:44:11,886 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-15 20:44:11,886 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-15 20:44:11,886 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Missing Rails.application.secrets.openid_connect_signing_key for production environment. The secret will be generated and stored in config/secrets.yml.
2018-07-15 20:44:29,929 INFO exited: sidekiq (exit status 1; not expected)
gitlabhq_production already exists
2018-07-15 20:44:30,933 INFO spawned: 'sidekiq' with pid 713
2018-07-15 20:44:31,935 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Clearing cache...
2018-07-15 20:44:50,993 INFO exited: sidekiq (exit status 1; not expected)
2018-07-15 20:44:51,996 INFO spawned: 'sidekiq' with pid 727
2018-07-15 20:44:52,999 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)```

Docker-compose used to spin up an instance also does not work.

version: '2'

services:
  redis:
    restart: always
    image: sameersbn/redis:latest
    command:
    - --loglevel warning
    volumes:
    - ./gitlab/redis:/var/lib/redis:Z

  postgresql:
    restart: always
    image: sameersbn/postgresql:9.6-2
    volumes:
    - ./gitlab/postgresql:/var/lib/postgresql:Z
    environment:
    - DB_USER=gitlab
    - DB_PASS=password
    - DB_NAME=gitlabhq_production
    - DB_EXTENSION=pg_trgm

  gitlab:
    restart: always
    image: sameersbn/gitlab:11.0.3
    depends_on:
    - redis
    - postgresql
    ports:
    - "10080:80"
    - "10022:22"
    volumes:
    - ./gitlab/gitlab:/home/git/data:Z
    environment:
    - DEBUG=false

    - DB_ADAPTER=postgresql
    - DB_HOST=postgresql
    - DB_PORT=5432
    - DB_USER=gitlab
    - DB_PASS=password
    - DB_NAME=gitlabhq_production

    - REDIS_HOST=redis
    - REDIS_PORT=6379

    - TZ=Asia/Kolkata
    - GITLAB_TIMEZONE=Kolkata

    - GITLAB_HTTPS=false
    - SSL_SELF_SIGNED=false

    - GITLAB_HOST=localhost
    - GITLAB_PORT=10080
    - GITLAB_SSH_PORT=10022
    - GITLAB_RELATIVE_URL_ROOT=
    - GITLAB_SECRETS_DB_KEY_BASE=V4VMktNNXWP9pp7Fmhb3b3jWKLgMcqXjJJgmqgdKHXntmCwCHgfsKjrTFgHkFp9F
    - GITLAB_SECRETS_SECRET_KEY_BASE=LVvLTCTdqd43rq7zw3Lr3gPtjRpqqf7PdtCCNthz79grFPJ4VqqkhR9pwgmbv9zn
    - GITLAB_SECRETS_OTP_KEY_BASE=pN3sCHgRh44JXcdHKzXHXrbXxPRx9dkCJHtmCVNvhgvr3XzNcXxspLc9Pmr7r4WL

    - GITLAB_ROOT_PASSWORD=
    - GITLAB_ROOT_EMAIL=

    - GITLAB_NOTIFY_ON_BROKEN_BUILDS=true
    - GITLAB_NOTIFY_PUSHER=false

    - [email protected]
    - [email protected]
    - [email protected]

    - GITLAB_BACKUP_SCHEDULE=daily
    - GITLAB_BACKUP_TIME=01:00

    - SMTP_ENABLED=false
    - SMTP_DOMAIN=www.example.com
    - SMTP_HOST=smtp.gmail.com
    - SMTP_PORT=587
    - [email protected]
    - SMTP_PASS=password
    - SMTP_STARTTLS=true
    - SMTP_AUTHENTICATION=login

    - IMAP_ENABLED=false
    - IMAP_HOST=imap.gmail.com
    - IMAP_PORT=993
    - [email protected]
    - IMAP_PASS=password
    - IMAP_SSL=true
    - IMAP_STARTTLS=false

    - OAUTH_ENABLED=false
    - OAUTH_AUTO_SIGN_IN_WITH_PROVIDER=
    - OAUTH_ALLOW_SSO=
    - OAUTH_BLOCK_AUTO_CREATED_USERS=true
    - OAUTH_AUTO_LINK_LDAP_USER=false
    - OAUTH_AUTO_LINK_SAML_USER=false
    - OAUTH_EXTERNAL_PROVIDERS=

    - OAUTH_CAS3_LABEL=cas3
    - OAUTH_CAS3_SERVER=
    - OAUTH_CAS3_DISABLE_SSL_VERIFICATION=false
    - OAUTH_CAS3_LOGIN_URL=/cas/login
    - OAUTH_CAS3_VALIDATE_URL=/cas/p3/serviceValidate
    - OAUTH_CAS3_LOGOUT_URL=/cas/logout

    - OAUTH_GOOGLE_API_KEY=
    - OAUTH_GOOGLE_APP_SECRET=
    - OAUTH_GOOGLE_RESTRICT_DOMAIN=

    - OAUTH_FACEBOOK_API_KEY=
    - OAUTH_FACEBOOK_APP_SECRET=

    - OAUTH_TWITTER_API_KEY=
    - OAUTH_TWITTER_APP_SECRET=

    - OAUTH_GITHUB_API_KEY=
    - OAUTH_GITHUB_APP_SECRET=
    - OAUTH_GITHUB_URL=
    - OAUTH_GITHUB_VERIFY_SSL=

    - OAUTH_GITLAB_API_KEY=
    - OAUTH_GITLAB_APP_SECRET=

    - OAUTH_BITBUCKET_API_KEY=
    - OAUTH_BITBUCKET_APP_SECRET=

    - OAUTH_SAML_ASSERTION_CONSUMER_SERVICE_URL=
    - OAUTH_SAML_IDP_CERT_FINGERPRINT=
    - OAUTH_SAML_IDP_SSO_TARGET_URL=
    - OAUTH_SAML_ISSUER=
    - OAUTH_SAML_LABEL="Our SAML Provider"
    - OAUTH_SAML_NAME_IDENTIFIER_FORMAT=urn:oasis:names:tc:SAML:2.0:nameid-format:transient
    - OAUTH_SAML_GROUPS_ATTRIBUTE=
    - OAUTH_SAML_EXTERNAL_GROUPS=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_EMAIL=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_NAME=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_FIRST_NAME=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_LAST_NAME=

    - OAUTH_CROWD_SERVER_URL=
    - OAUTH_CROWD_APP_NAME=
    - OAUTH_CROWD_APP_PASSWORD=

    - OAUTH_AUTH0_CLIENT_ID=
    - OAUTH_AUTH0_CLIENT_SECRET=
    - OAUTH_AUTH0_DOMAIN=

    - OAUTH_AZURE_API_KEY=
    - OAUTH_AZURE_API_SECRET=
    - OAUTH_AZURE_TENANT_ID=

Most helpful comment

+1 for this solution.
In docker-compose.override.yml I changed the REDIS to this:

redis:
    restart: always
    image: sameersbn/redis:latest
    command:
    - --loglevel warning
    - --protected-mode no
    volumes:
    - /srv/docker/gitlab/redis:/var/lib/redis:Z

The change is in "protected-mode no".

All 7 comments

I've tried on a dedicated server that has 16 gigs of ram and 16 gigs of swap. I've tried my configuration via docker compose and I've tried just running the commands shown in the README and configuring what's supposed to be configured by recommendation of the Quick Start all to no avail. This container looks like the best/most complete and I need it to be working by end of day tomorrow for a talk I'm giving. Is there any chance that someone can explain why this doesn't work out of the box at all?

I also tried it on an OS X machine with 32 gigs of ram also with no success.

@chamunks I had to make the following changes to get around this issue.

`
12:29 $ git diff
diff --git a/docker-compose.yml b/docker-compose.yml
index d180dfe..dd7b7db 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -5,7 +5,7 @@ services:
restart: always
image: sameersbn/redis:latest
command:
- - --loglevel warning
+ - --loglevel warning --protected-mode no
volumes:
- /srv/docker/gitlab/redis:/var/lib/redis:Z

@@ -22,7 +22,7 @@ services:

gitlab:
restart: always
- image: sameersbn/gitlab:11.0.3
+ image: sameersbn/gitlab:10.7.4
depends_on:
- redis
- postgresql
✔ ~/git/docker-gitlab [master|✚ 1]
`

+1 for this solution.
In docker-compose.override.yml I changed the REDIS to this:

redis:
    restart: always
    image: sameersbn/redis:latest
    command:
    - --loglevel warning
    - --protected-mode no
    volumes:
    - /srv/docker/gitlab/redis:/var/lib/redis:Z

The change is in "protected-mode no".

@keyj I'm not sure where to find docker-compose.override.yml but I've made the change and I'm still getting the following.

_1      | 2018-07-22 23:03:26,381 WARN Included extra file "/etc/supervisor/conf.d/sidekiq.conf" during parsing
gitlab_1      | 2018-07-22 23:03:26,381 WARN Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing
gitlab_1      | 2018-07-22 23:03:26,381 WARN Included extra file "/etc/supervisor/conf.d/unicorn.conf" during parsing
gitlab_1      | 2018-07-22 23:03:26,392 INFO RPC interface 'supervisor' initialized
gitlab_1      | 2018-07-22 23:03:26,392 CRIT Server 'unix_http_server' running without any HTTP authentication checking
gitlab_1      | 2018-07-22 23:03:26,392 INFO supervisord started with pid 558
gitlab_1      | 2018-07-22 23:03:27,395 INFO spawned: 'gitaly' with pid 575
gitlab_1      | 2018-07-22 23:03:27,397 INFO spawned: 'sidekiq' with pid 576
gitlab_1      | 2018-07-22 23:03:27,399 INFO spawned: 'unicorn' with pid 577
gitlab_1      | 2018-07-22 23:03:27,401 INFO spawned: 'gitlab-workhorse' with pid 578
gitlab_1      | 2018-07-22 23:03:27,403 INFO spawned: 'cron' with pid 583
gitlab_1      | 2018-07-22 23:03:27,405 INFO spawned: 'nginx' with pid 584
gitlab_1      | 2018-07-22 23:03:27,407 INFO spawned: 'sshd' with pid 587
gitlab_1      | 2018-07-22 23:03:28,458 INFO success: gitaly entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2018-07-22 23:03:28,458 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2018-07-22 23:03:28,458 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2018-07-22 23:03:28,458 INFO success: gitlab-workhorse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2018-07-22 23:03:28,458 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2018-07-22 23:03:28,458 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2018-07-22 23:03:28,458 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2018-07-22 23:03:28,548 INFO exited: unicorn (exit status 1; not expected)
gitlab_1      | 2018-07-22 23:03:28,558 INFO spawned: 'unicorn' with pid 619
gitlab_1      | 2018-07-22 23:03:29,560 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2018-07-22 23:03:30,189 INFO exited: unicorn (exit status 1; not expected)
gitlab_1      | 2018-07-22 23:03:31,192 INFO spawned: 'unicorn' with pid 701
gitlab_1      | 2018-07-22 23:03:32,170 INFO exited: unicorn (exit status 1; not expected)
gitlab_1      | 2018-07-22 23:03:33,173 INFO spawned: 'unicorn' with pid 704
gitlab_1      | 2018-07-22 23:03:34,142 INFO exited: unicorn (exit status 1; not expected)
gitlab_1      | 2018-07-22 23:03:36,146 INFO spawned: 'unicorn' with pid 707
gitlab_1      | 2018-07-22 23:03:37,151 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2018-07-22 23:03:37,209 INFO exited: unicorn (exit status 1; not expected)
gitlab_1      | 2018-07-22 23:03:38,212 INFO spawned: 'unicorn' with pid 710
gitlab_1      | 2018-07-22 23:03:39,214 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2018-07-22 23:03:39,366 INFO exited: unicorn (exit status 1; not expected)
gitlab_1      | 2018-07-22 23:03:40,369 INFO spawned: 'unicorn' with pid 713
gitlab_1      | 2018-07-22 23:03:41,372 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2018-07-22 23:03:41,377 INFO exited: unicorn (exit status 1; not expected)
gitlab_1      | 2018-07-22 23:03:42,381 INFO spawned: 'unicorn' with pid 716
gitlab_1      | 2018-07-22 23:03:43,367 INFO exited: unicorn (exit status 1; not expected)
gitlab_1      | 2018-07-22 23:03:44,371 INFO spawned: 'unicorn' with pid 725
gitlab_1      | 2018-07-22 23:03:45,373 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2018-07-22 23:03:45,381 INFO exited: unicorn (exit status 1; not expected)
gitlab_1      | 2018-07-22 23:03:46,384 INFO spawned: 'unicorn' with pid 728

I'm attempting to execute with the following docker-compose.yml

version: "2"

networks:
  traefik:
    external: true

services:
#####################
## Redis Container ##
#####################
  redisio:
    image: 'sameersbn/redis:latest'
    restart: always
    labels:
      - traefik.enable=false
    command:
    - --loglevel warning
    - --protected-mode no
    volumes:
    - ./data/redis:/var/lib/redis:Z
    expose:
      - '6379'
    networks:
      traefik:
        ipv4_address: 10.5.0.62
########################
## Postgres Container ##
########################
  postgresql:
    image: postgres:alpine
    restart: always
    environment:
      POSTGRES_PASSWORD: REDACTED
      ## Might only work with the sameersbn postgres container.
      DB_EXTENSION: "pg_trgm"
    volumes:
      - ./data/db:/var/lib/postgresql/data
    labels:
      - traefik.enable=false
    networks:
      traefik:
        ipv4_address: 10.5.0.61
############
## GitLab ##
############
  gitlab:
    ## Something other than this
    ## https://github.com/sameersbn/docker-gitlab/blob/master/docker-compose.yml
    ## Documentation
    ## https://hub.docker.com/r/sameersbn/gitlab/
    image: sameersbn/gitlab:11.0.3
    depends_on:
      - redisio
      - postgresql
    restart: always
    labels:
      # https://docs.traefik.io/user-guide/docker-and-lets-encrypt/
      - "traefik.gitlab.backend=gitlab"
      - "traefik.registry.backend=registry"
      - "traefik.docker.network=traefik"
      - "traefik.gitlab.frontend.rule=Host:gitlab.REDACTED"
      - "traefik.registry.frontend.rule=Host:registry.REDACTED"
      - "traefik.gitlab.expose=true"
      - "traefik.registry.expose=true"
      - "traefik.gitlab.port=80"
      - "traefik.registry.port=8181"
      ## An example I pulled from a github issue somewhere.
      # - 'traefik.s1.frontend.rule=Host:s1.mydomain.org'
      # - "traefik.s1.port=80"
      # - 'traefik.s2.frontend.rule=Host:s2.mydomain.org'
      # - "traefik.s2.port=8081"
    # links:
    #   - 'gitlab-postgresql:postgresql'
    #   - 'gitlab-redis:redisio'
    expose:
      ## Expose for Traefik
      - '80'
      - '8181'
    ports:
      ## Port bind for git
      - '10022:22'
    networks:
      traefik:
        ipv4_address: 10.5.0.60
    environment:
      - DEBUG=true

      - DB_ADAPTER=postgresql
      - DB_HOST=postgresql
      - DB_PORT=5432
      - DB_USER=postgres
      - DB_PASS=REDACTED
      - DB_NAME=postgres

      - REDIS_HOST=redisio
      - REDIS_PORT=6379

      - TZ=America/Toronto
      - GITLAB_TIMEZONE=Toronto

      - GITLAB_HTTPS=false
      - SSL_SELF_SIGNED=false

      - GITLAB_HOST=localhost
      - GITLAB_PORT=443
      - GITLAB_SSH_PORT=10022
      - GITLAB_RELATIVE_URL_ROOT=
      - GITLAB_SECRETS_DB_KEY_BASE=REDACTED
      - GITLAB_SECRETS_SECRET_KEY_BASE=REDACTED
      - GITLAB_SECRETS_OTP_KEY_BASE=REDACTED

      - GITLAB_ROOT_PASSWORD=
      - [email protected]

      - GITLAB_NOTIFY_ON_BROKEN_BUILDS=true
      - GITLAB_NOTIFY_PUSHER=false

      - [email protected]
      - [email protected]
      - GITLAB_INCOMING_EMAIL_ADDRESS='gitlab+%{key}@gitlab.REDACTED'

      - GITLAB_BACKUP_SCHEDULE=daily
      - GITLAB_BACKUP_TIME=01:00

      - SMTP_ENABLED=false
      - SMTP_DOMAIN=REDACTED
      - SMTP_HOST=REDACTED
      - SMTP_PORT=587
      - [email protected]
      - SMTP_PASS=REDACTED
      - SMTP_STARTTLS=true
      - SMTP_AUTHENTICATION=login

      - IMAP_ENABLED=false
      - IMAP_HOST=REDACTED
      - IMAP_PORT=993
      - [email protected]
      - IMAP_PASS=REDACTED
      - IMAP_SSL=true
      - IMAP_STARTTLS=false

      - OAUTH_ENABLED=false
    volumes:
      - './data/gitlab:/home/git/data'

@keyj thanks for enlightening me with the new docker-compose.override.yml feature I don't entirely plan on leaving it as a fully vanilla configuration and want to go something a bit more like this direction if possible. However, I can't seem to get it to a point where its content with what I have.

Going with the method that @keyj posted I've managed to get the baseline working. I guess at this point the next step is to spam the crap out of the overrides instead of making my own from scratch unless someone can see what I'm missing in mine and suggest an edit or two.

@keyj thanks for the tip on the --protected-mode no flag. I'll send in a PR to use it with the latest redis release. Thanks

Was this page helpful?
0 / 5 - 0 ratings

Related issues

STOIE picture STOIE  Â·  4Comments

tomastzn picture tomastzn  Â·  4Comments

lenovouser picture lenovouser  Â·  3Comments

rakesh-k picture rakesh-k  Â·  3Comments

tarach picture tarach  Â·  5Comments