Fix: systemctl Service Failed to Start – Unit Entered Failed State

The Error

You try to start a service with systemctl and it fails:

$ sudo systemctl start myapp
Job for myapp.service failed because the control process exited with error code.
See "systemctl status myapp.service" and "journalctl -xe" for details.

When you check the status, you see one of these:

● myapp.service - My Application
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled)
     Active: failed (Result: exit-code) since Mon 2026-06-01 14:23:01 UTC; 5s ago
    Process: 48291 ExecStart=/usr/local/bin/myapp (code=exited, status=203/EXEC)

myapp.service: Main process exited, code=exited, status=1/FAILURE
myapp.service: Failed with result 'exit-code'.

myapp.service: Start request repeated too quickly.
myapp.service: Failed with result 'start-limit-hit'.

myapp.service: Unit entered failed state.

All of these mean systemd tried to start your service, the process either never launched or exited immediately, and systemd marked the unit as failed.

Why This Happens

systemd is the init system on virtually every modern Linux distribution. When you define a service (a .service unit file), systemd is responsible for starting the process, monitoring it, restarting it if it crashes, and managing its dependencies.

A service enters the “failed” state when systemd cannot successfully start and keep the process running. The root causes fall into a few categories:

• The binary path is wrong. The ExecStart directive points to a file that doesn’t exist or isn’t executable.
• File permissions. The service user doesn’t have permission to read the binary, write to log files, or access required directories.
• Missing dependencies. The service starts before a resource it needs (database, network, mounted filesystem) is available.
• Environment variables are missing. The app expects environment variables that exist in your shell but not in the systemd environment.
• Port conflicts. Another process already occupies the port your service needs.
• The process exits immediately. The application crashes on startup due to bad configuration, missing libraries, or runtime errors.
• Start rate limiting. The service crashed and restarted too many times in a short period, and systemd stopped trying.

The first step in every case is reading the logs.

Fix 1: Read the Logs with journalctl

Before changing anything, find out what actually went wrong. journalctl is the single most important tool for debugging systemd service failures:

# Show logs for the specific service
sudo journalctl -u myapp.service -n 50 --no-pager

This shows the last 50 log lines for myapp.service. Look for error messages from your application or from systemd itself.

For more context, show logs since the last boot:

sudo journalctl -u myapp.service -b --no-pager

To follow logs in real time while you try to start the service in another terminal:

sudo journalctl -u myapp.service -f

Then in another terminal:

sudo systemctl start myapp

You can also use systemctl status for a quick summary:

sudo systemctl status myapp.service

This shows the last few log lines, the PID, the exit code, and the current state. The exit status code is critical:

For a deeper timeline analysis across all services during boot, use:

systemd-analyze blame
systemd-analyze critical-chain myapp.service

This shows how long each service took to start and what was blocking your service in the dependency chain.

Fix 2: Fix the ExecStart Path

Exit status 203/EXEC almost always means the path in ExecStart is wrong. This is the most common systemd service error.

Check your unit file:

sudo systemctl cat myapp.service

Look at the ExecStart line:

[Service]
ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.yaml

Verify the binary exists and is executable:

ls -la /usr/local/bin/myapp
which myapp

Common mistakes:

The binary was installed somewhere else. If you installed with pip, npm, cargo, or a language-specific package manager, the binary may be in a user-local path like /home/deploy/.local/bin/ or /home/deploy/.nvm/versions/node/v20.11.0/bin/node. systemd does not load your shell profile, so $PATH expansion does not work in unit files. Always use absolute paths.

The binary is a script without a shebang. If ExecStart points to a Python or shell script, make sure the file has a proper shebang (#!/usr/bin/env python3) and is executable (chmod +x). For more on script permission issues, see Fix: bash: Permission denied.

The binary was deleted or moved after a deployment. If you deploy by replacing files, the old binary may be gone. Rebuild or redeploy.

After fixing the path, reload and restart:

sudo systemctl daemon-reload
sudo systemctl start myapp

You must run daemon-reload every time you edit a unit file. systemd caches unit file contents and will not pick up changes without it.

Fix 3: Fix File Permissions and Ownership

Your service binary exists, but the user systemd runs the service as cannot access it. Check which user the service runs as:

sudo systemctl cat myapp.service | grep -E "^User=|^Group="

If no User= is specified, the service runs as root. If a user is specified, verify that user exists:

id myapp

If the user doesn’t exist (status 217/USER), create it:

sudo useradd -r -s /sbin/nologin myapp

Then make sure the service user can access everything it needs:

# The binary itself
sudo chmod 755 /usr/local/bin/myapp

# Configuration files
sudo chown -R myapp:myapp /etc/myapp/
sudo chmod 640 /etc/myapp/config.yaml

# Data and log directories
sudo mkdir -p /var/lib/myapp /var/log/myapp
sudo chown -R myapp:myapp /var/lib/myapp /var/log/myapp

systemd also offers built-in sandboxing directives that restrict filesystem access. If your unit file uses ProtectSystem=, ReadOnlyPaths=, or PrivateTmp=, these can prevent your service from writing to paths it needs:

[Service]
# This makes /usr, /boot, /efi read-only
ProtectSystem=full

# Explicitly allow writing to specific paths
ReadWritePaths=/var/lib/myapp /var/log/myapp

If you’re not sure whether sandboxing is blocking access, temporarily remove the Protect* and ReadOnly* directives, restart, and see if the service starts.

Fix 4: Fix Missing Dependencies with After= and Requires=

Your service starts before a resource it depends on is ready. This is extremely common with applications that need a database or network connection at startup.

myapp.service: Main process exited, code=exited, status=1/FAILURE

And in the logs:

Error: connect ECONNREFUSED 127.0.0.1:5432

The application tried to connect to PostgreSQL, but PostgreSQL hadn’t started yet. Fix this by declaring the dependency:

[Unit]
Description=My Application
After=network.target postgresql.service
Requires=postgresql.service

• After= means “start my service after these units.” This controls ordering but does not start the dependency.
• Requires= means “if this dependency fails or is stopped, stop my service too.” It also ensures the dependency is started.
• Wants= is a softer version of Requires=. The dependency is started but if it fails, your service is not affected.

Common dependency targets:

If your app connects to a database on startup and crashes when it’s unavailable, you should also make the application resilient by adding connection retry logic. The After= directive guarantees ordering but not that the database is ready to accept connections. For related database connectivity issues, see Fix: PostgreSQL Connection Refused.

Pro Tip: Use systemd-analyze plot > boot.svg to generate a visual timeline of your system boot. Open the SVG in a browser to see exactly when each service starts relative to others. This instantly reveals ordering problems you’d never catch from log files alone.

Fix 5: Fix Environment Variables

Your application works fine when you run it manually but fails under systemd because the environment is completely different. systemd does not source ~/.bashrc, ~/.profile, or any shell startup files.

Check what environment your service actually sees:

sudo systemctl show myapp.service -p Environment

There are three ways to provide environment variables to a systemd service:

Option 1: Inline in the unit file (for a few variables):

[Service]
Environment="NODE_ENV=production"
Environment="PORT=3000"
Environment="DATABASE_URL=postgresql://user:pass@localhost/mydb"

Option 2: Environment file (for many variables or secrets):

[Service]
EnvironmentFile=/etc/myapp/.env

The file format is one KEY=VALUE per line, no export, no quotes needed:

NODE_ENV=production
PORT=3000
DATABASE_URL=postgresql://user:pass@localhost/mydb
SECRET_KEY=abc123

Set proper permissions on the file so only root and the service user can read it:

sudo chown root:myapp /etc/myapp/.env
sudo chmod 640 /etc/myapp/.env

Option 3: PassEnvironment (pass specific variables from the systemd manager):

[Service]
PassEnvironment=HOME LANG

This is rarely needed. Option 2 (EnvironmentFile) is the standard approach for production services.

After changing environment settings:

sudo systemctl daemon-reload
sudo systemctl restart myapp

Fix 6: Fix Port Conflicts

Your service tries to bind to a port that another process already occupies. The journal logs will show something like:

Error: listen EADDRINUSE: address already in use :::3000

Or:

bind() to 0.0.0.0:8080 failed (98: Address already in use)

Find out what’s using the port:

sudo ss -tlnp | grep :3000

This shows the process ID and name. You then need to either stop the conflicting process, change the port your service uses, or kill the stale process.

If the previous instance of your own service didn’t shut down cleanly and is still holding the port, check for zombie processes:

ps aux | grep myapp

Kill any leftover processes:

sudo kill <PID>

Then start the service again. If you’re running multiple services behind Nginx and encounter upstream connection issues from port conflicts, see Fix: Nginx 502 Bad Gateway.

For a longer-term fix, make sure your service unit file has a clean shutdown mechanism:

[Service]
ExecStop=/bin/kill -SIGTERM $MAINPID
TimeoutStopSec=10

This gives your process 10 seconds to shut down gracefully before systemd sends SIGKILL.

Fix 7: Fix Restart Policies and Rate Limiting

When a service repeatedly fails, systemd stops trying to restart it to prevent resource exhaustion:

myapp.service: Start request repeated too quickly.
myapp.service: Failed with result 'start-limit-hit'.

This means the service crashed and restarted more than the allowed number of times within the rate-limiting window. By default, systemd allows 5 start attempts within 10 seconds.

First, fix the underlying reason the service keeps crashing (use journalctl -u myapp.service to find out). Then reset the failure counter:

sudo systemctl reset-failed myapp.service
sudo systemctl start myapp

If your service legitimately needs time to stabilize (perhaps it connects to an external dependency that’s flaky during boot), you can adjust the restart behavior:

[Service]
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=5

• Restart=on-failure restarts the service only when it exits with a non-zero exit code (not on clean shutdown).
• RestartSec=5 waits 5 seconds between restart attempts. This prevents the rapid restart loop that triggers rate limiting.
• StartLimitIntervalSec=60 sets the window to 60 seconds (instead of the default 10).
• StartLimitBurst=5 allows 5 start attempts within that window.

Note that StartLimitIntervalSec and StartLimitBurst belong in the [Unit] section on older systemd versions (before 230), but in the [Service] section on newer versions. If one placement doesn’t work, try the other.

Common Mistake: Setting RestartSec=0 seems like a good idea for fast recovery, but it’s what causes the “start request repeated too quickly” error in the first place. A service that crashes immediately and restarts with no delay will hit the rate limit within a second. Always set RestartSec to at least 1-2 seconds.

Fix 8: Fix SELinux Denials

On RHEL, CentOS, Fedora, and Amazon Linux, SELinux can prevent your service from starting even when everything else is correct. The journal won’t always make this obvious — you might just see a generic exit code 1.

Check for SELinux denials:

sudo ausearch -m avc -ts recent --comm myapp

Or search the audit log:

sudo grep denied /var/log/audit/audit.log | tail -20

Temporarily set SELinux to permissive to confirm it’s the cause:

sudo setenforce 0
sudo systemctl start myapp
# If it works, SELinux was blocking it
sudo setenforce 1

For a permanent fix, generate a custom policy module from the denials:

sudo ausearch -m avc -ts recent | audit2allow -M myapp-policy
sudo semodule -i myapp-policy.pp

Common SELinux booleans for services:

# Allow a service to bind to non-standard ports
sudo semanage port -a -t http_port_t -p tcp 3000

# Allow a service to connect to the network
sudo setsebool -P httpd_can_network_connect 1

# Allow a service to use the database
sudo setsebool -P httpd_can_network_connect_db 1

For scripts that run as part of a service, you may also need to set the correct file context:

sudo semanage fcontext -a -t bin_t "/usr/local/bin/myapp"
sudo restorecon -v /usr/local/bin/myapp

Fix 9: Fix User and Group Settings

If your unit file specifies a User= or Group= that doesn’t exist, systemd fails immediately with status 217/USER:

myapp.service: Failed at step USER spawning /usr/local/bin/myapp: No such process

Verify the user exists:

id myapp

Create a system user for the service if needed:

sudo useradd -r -s /sbin/nologin -d /var/lib/myapp myapp

The -r flag creates a system user (low UID, no home directory created by default on some distros, no aging). The -s /sbin/nologin prevents interactive login for security.

If your service needs to access files owned by a specific group (for example, reading TLS certificates that are group-readable by the ssl-cert group):

[Service]
User=myapp
Group=myapp
SupplementaryGroups=ssl-cert

When switching from running a service as root to a dedicated user, remember to update ownership of all data directories, log files, PID files, and Unix sockets that the service creates. Missing even one path will cause a permission failure on startup. For SSH-related permission problems on the server side, see Fix: SSH Connection Timed Out.

Fix 10: Fix Socket Activation Issues

systemd can manage sockets on behalf of your service, starting the service on demand when a connection arrives. This is configured with a .socket unit:

# /etc/systemd/system/myapp.socket
[Socket]
ListenStream=3000
Accept=no

[Install]
WantedBy=sockets.target

If socket activation isn’t working correctly:

Check both the socket and service status:

sudo systemctl status myapp.socket
sudo systemctl status myapp.service

The socket and service names must match. If your socket is myapp.socket, systemd looks for myapp.service by default. To override this, use Service= in the socket unit.

Your application must accept the socket file descriptor. systemd passes the listening socket as file descriptor 3 (not by creating a new one). If your app creates its own socket, it conflicts with the one systemd manages.

For Node.js with socket activation:

const http = require('http');
const server = http.createServer(app);

// Use the socket passed by systemd (fd 3) if available
if (process.env.LISTEN_FDS) {
  server.listen({ fd: 3 });
} else {
  server.listen(3000);
}

If you don’t need socket activation, make sure you’re starting the .service directly and not relying on a .socket unit that doesn’t match your application’s expectations:

sudo systemctl stop myapp.socket
sudo systemctl disable myapp.socket
sudo systemctl start myapp.service

Fix 11: Debug with systemd-analyze

When you’ve checked the obvious causes and the service still fails, use systemd-analyze to dig deeper:

Verify the unit file syntax:

systemd-analyze verify /etc/systemd/system/myapp.service

This catches syntax errors, unknown directives, and missing dependencies without actually starting the service.

Check the security posture of the service:

systemd-analyze security myapp.service

This scores your service from 0 (most exposed) to 10 (most locked down) and lists all the sandboxing directives. It also reveals which restrictions are active, which can help identify why a service is being blocked from accessing resources.

Run the service interactively to see exactly what happens:

sudo systemd-run --scope --uid=myapp /usr/local/bin/myapp --config /etc/myapp/config.yaml

This runs the binary under a transient systemd scope with the same user as your service, but with output going directly to your terminal. Any crash messages or missing library errors will be immediately visible. This is often the fastest way to debug when journalctl output is unclear.

Still Not Working?

The service works manually but fails under systemd

Run the command exactly as systemd would, with a clean environment:

sudo -u myapp env -i PATH=/usr/local/bin:/usr/bin:/bin /usr/local/bin/myapp

The env -i clears all environment variables. If the app fails here but works when you run it as your own user, the problem is environment variables, PATH, or file permissions relative to the service user.

Missing shared libraries

The application might depend on shared libraries installed in a non-standard path:

ldd /usr/local/bin/myapp

If any line says “not found,” the library is missing. Install it or tell the dynamic linker where to find it:

echo "/opt/myapp/lib" | sudo tee /etc/ld.so.conf.d/myapp.conf
sudo ldconfig

Working directory doesn’t exist

Status 200/CHDIR means the WorkingDirectory path doesn’t exist:

[Service]
WorkingDirectory=/opt/myapp

Create it:

sudo mkdir -p /opt/myapp
sudo chown myapp:myapp /opt/myapp

Docker Compose services managed by systemd

If you’re wrapping docker compose up in a systemd service, make sure the unit file handles Docker’s own startup and shutdown correctly. For issues with docker compose up itself, see Fix: Docker Compose Up Errors:

[Unit]
Description=My Docker Compose Application
After=docker.service network-online.target
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=120

[Install]
WantedBy=multi-user.target

The Type=oneshot with RemainAfterExit=yes is critical here. docker compose up -d exits immediately after starting the containers, so systemd needs to know that the exit is expected and the service should still be considered “active.”

Checking for masked or aliased units

If systemctl start silently does nothing or gives a cryptic error, the unit might be masked:

sudo systemctl status myapp.service

If it says Loaded: masked, the unit has been explicitly disabled:

sudo systemctl unmask myapp.service
sudo systemctl start myapp.service

Last resort: enable debug logging for systemd

If nothing else reveals the problem, enable debug-level logging for the systemd manager:

sudo systemd-analyze set-log-level debug
sudo systemctl restart myapp
sudo journalctl -u myapp.service -b --no-pager
# Reset when done
sudo systemd-analyze set-log-level info

This produces extremely verbose output but can reveal issues with namespace setup, cgroup assignment, capability dropping, and other low-level operations that aren’t logged at the default level.

Related: Fix: Nginx 502 Bad Gateway | Fix: bash: Permission denied | Fix: PostgreSQL Connection Refused | Fix: Docker Compose Up Errors | Fix: SSH Connection Timed Out