Troubleshooting

The errors below occur during the Linux package installation and the initial configuration of the instance. If your case is not here, collect the output of the commands from the General diagnostics section and contact the vendor.

Application does not start: the libcrypt.so.1 library is missing

Where it appears. RED OS, the output of the package installation and the initial configuration:

libcrypt.so.1: cannot open shared object file

Cause. The minimal RED OS installation does not contain the libcrypt.so.1 library.

Action. Install the package with the library and repeat the configuration:

sudo dnf install -y libxcrypt-compat
sudo gitlab-ctl reconfigure

Configuration fails with an SELinux error

Where it appears. RED OS, the gitlab-ctl reconfigure run, the bash[Set proper security context on ssh files for selinux] resource:

ValueError: Type gitlab_shell_t is invalid
semanage: command not found

Cause. The server has no SELinux utilities from the policycoreutils-python-utils package, or the policy module was not loaded during the initial configuration.

Action. Install the utilities, repeat the configuration and check that the policy module is loaded:

sudo dnf install -y policycoreutils-python-utils
sudo gitlab-ctl reconfigure
sudo semodule -l | grep -i gitlab

Instance name is unreachable: curl returns 000

Cause. The <HOSTNAME> name does not resolve on the server or on the machine that opens the web interface, or port 80 or 443 is unreachable: closed by the firewall or nginx is not running.

Action.

1. Check the name with the getent hosts <HOSTNAME> command. If the name is missing, create an A or AAAA DNS record pointing to the IP address of the machine, or add a line to /etc/hosts on a stand without DNS. The name must match external_url and the CN and SAN fields of the certificate.

2. If the name resolves to the expected IP address, check the state of nginx and the listening ports on the server:

   sudo gitlab-ctl status nginx
   sudo ss -tlnp | grep -E ':(80|443)\b'

3. Check the HTTP entry point from the workstation:

   curl -sS -o /dev/null -w "HTTP %{http_code}\n" \
     http://<HOSTNAME>/

   Any HTTP code (usually 301 or 302 once TLS is enabled) means that nginx is reachable. Code 000 means that the connection was not established. If nginx is running and the port is closed from the outside, open it as described in Network access (firewall).

Web interface returns 502 after configuration or restart

Cause. The application is still starting: during the first minute or two after reconfigure or restart, code 502 is expected.

Action. Wait and check the state of the services with the sudo gitlab-ctl status command. If 502 lasts longer than a few minutes, read the log with sudo gitlab-ctl tail puma: a frequent cause is a lack of memory, see Resources.

Configuration fails with a Let’s Encrypt error

Where it appears. The gitlab-ctl reconfigure run after an https:// address has been set in external_url while the certificate files are not on disk.

Cause. Without a certificate on disk, obtaining a certificate through Let’s Encrypt is turned on, which is impossible in an isolated network.

Action. Place the certificate and set letsencrypt['enable'] = false as described in TLS, then run sudo gitlab-ctl reconfigure.

Reading an error in the reconfigure output

Where it appears. The output of a gitlab-ctl reconfigure command that ended with an error.

Action. At the end of the output, find the block with the resource name and the cause:

Error executing action ... on resource '...'

The full run log is kept in the /var/log/gitlab/reconfigure/ directory.

General diagnostics

These commands apply to any of the errors listed above:

# The state of every service.
sudo gitlab-ctl status
# The live log of a service, for example puma or nginx.
sudo gitlab-ctl tail <SERVICE>
# The instance self-check.
sudo gitlab-rake gitlab:check SANITIZE=true
# The log directories per service.
ls /var/log/gitlab/