Troubleshooting the Boundary Area Where Ping Succeeds but TCP Connection Fails

Fri, 09 Oct 2026 10:46:33 +0900

In infrastructure operations, the issue where “Ping succeeds, but connections to the server (such as SSH or HTTP) fail” frequently occurs. Relying solely on ICMP for L3-level reachability verification while neglecting the verification of L4 transport layer or L7 application layer leads to delayed incident recovery and incorrect root cause identification.

The ICMP echo test indicates a successful state with low latency and 0% packet loss.

$ ping 192.168.10.100
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.
64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.3 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.2 ms

However, initiating an SSH session or HTTP request results in a connection failure.

$ ssh 192.168.10.100
ssh: connect to host 192.168.10.100 port 22: Connection refused

Alternatively, a timeout occurs during the connection attempt.

$ curl http://192.168.10.100:8080
curl: (7) Failed to connect to 192.168.10.100 port 8080: Connection timed out

The assumption that “since Ping succeeded, the network is normal, and connection failure means the server itself is broken” is technically incorrect. This is because ICMP operates at the network layer (L3), whereas key application protocols such as SSH, HTTP, HTTPS, PostgreSQL, MySQL, and Redis rely on transport layer (L4) TCP sockets bound to specific port numbers.

This article systematically breaks down this boundary gap and establishes diagnostic methodologies centered on three core Linux utilities: ss, nc, and tcpdump.

1. Network Layer Abstraction and Service Mapping

💡 To effectively identify communication bottlenecks, target services are mapped to each layer of the OSI reference model and transport mechanisms.

ICMP: Used exclusively for diagnostic and control messaging such as ping (no transport port exists).
SSH: TCP port 22
HTTP: TCP port 80
HTTPS: TCP port 443
PostgreSQL: TCP port 5432
MySQL / MariaDB: TCP port 3306
Redis: TCP port 6379

Because ICMP operates independently of L4 TCP listening sockets, receiving a response to an ICMP Echo Request provides no guarantee that a TCP port is open or reachable.

2. Low-Layer Mechanics of `ping`

Executing ping 192.168.10.100 sends an ICMP Echo Request packet (Type 8) to the destination host, which returns an ICMP Echo Reply packet (Type 0).

+------------------------+ +------------------------+
| Client: 192.168.10.50 | | Server: 192.168.10.100 |
+------------------------+ +------------------------+
 | |
 |------------ ICMP Echo Request (Type 8) -------------&gt;|
 | |
 |&lt;----------- ICMP Echo Reply (Type 0) ----------------|
 | |

A successful Ping merely confirms that the path from the client through local routing, gateways, the physical network, and the target IP stack back to the ICMP reply is functional. It cannot verify whether firewall rules allow TCP frames or whether an application daemon on the host is actively processing sockets.

3. Structure of the TCP 3-Way Handshake

When initiating a TCP-based service connection, the operating system kernel must first establish a stateful connection via the TCP 3-Way Handshake.

Client (192.168.10.50) Server (192.168.10.100:22)
 | |
 |----------------------- 1. SYN ------------------------------&gt;|
 | |
 |&lt;------------------- 2. SYN-ACK ------------------------------|
 | |
 |----------------------- 3. ACK ------------------------------&gt;|
 | |
[ ESTABLISHED ] [ ESTABLISHED ]

SYN: The client sends a TCP frame with the SYN flag set to request connection initiation.
SYN-ACK: Upon receiving the SYN, the server checks the socket listening on port 22, allocates a connection state buffer, and responds with SYN-ACK.
ACK: The client responds with ACK, and both endpoints transition to the ESTABLISHED state.

For application protocols to function, this L4 sequence must complete successfully.

4. Diagnosing “Ping Succeeds + TCP Connection Fails”

Define the initial diagnostic state.

$ ping 192.168.10.100
# Success (0% packet loss)

$ nc -zv 192.168.10.100 22
# Failure

This discrepancy allows immediate narrowing down of the failure domain. Functional ICMP indicates that L3 routing and physical links are operational, while TCP failure indicates an issue with L4 sockets, firewalls, or service daemons.

5. Transport Layer Probing with `nc` (Netcat)

Netcat (nc) functions as a TCP and UDP connection analyzer. Its primary use case in production environments is verifying raw TCP reachability to a specific socket address.

To run Netcat, specify options for port scanning.

$ nc -zv 192.168.10.100 22

-z`: Zero-I/O mode (scans for listening daemons without sending payload data).

-v: Verbose output mode.

Determine network status from connection test execution results.

Handshake Success:

Connection to 192.168.10.100 22 port [tcp/ssh] succeeded!

Failure Pattern A:

nc: connect to 192.168.10.100 port 22 failed: Connection refused

Failure Pattern B:

nc: connect to 192.168.10.100 port 22 failed: Connection timed out

Understanding the distinction between Connection Refused and Connection Timed Out is critical in identifying root causes.

6. Mechanisms of Connection Refused and Connection Timeout

Mechanics of Connection Refused

⚠️ A Connection refused error indicates that the client’s SYN packet reached the destination host’s kernel, but the host explicitly rejected the connection and returned a TCP Reset (RST) frame.

Client Server
 | |
 |--------------------------- SYN ----------------------------&gt;| (Destination host reached)
 | | (No listening process)
 |&lt;------------------------ TCP RST ---------------------------|

Primary Causes:

Service Daemon Stopped: The process (sshd, nginx, mysqld, etc.) is not running.
Incorrect Service Port: The application is listening on a non-default port (e.g., 2222 instead of 22).
Bound Interface Mismatch: The daemon is bound strictly to 127.0.0.1 rather than 0.0.0.0.
Active Packet Filtering: The localhost firewall is configured to explicitly return TCP Reset packets (REJECT) rather than dropping them (DROP).

Mechanics of Connection Timeout

A Connection timed out error occurs when the client sends a SYN packet, but no response whatsoever is returned before the socket timeout threshold expires.

Client Network / Firewall / Target
 | |
 |--------------------------- SYN ----------------------------&gt;|
 | | ===&gt; [ Drop / Block ]
 | (No Response) |
 | ... Timeout ... |

Primary Causes:

Network Security Drop: Cloud security groups, network ACLs, or hardware firewalls are silently dropping inbound traffic.
Host-Level Firewall Drop: iptables, firewalld, or nftables are configured with a DROP action.
Routing Failure: Incorrect default routing or missing static routes at intermediate network switches or routers.
Asymmetric Routing / Return Path Failure: The server receives the SYN and sends a SYN-ACK, but the packet is dropped or misrouted on the return path.

7. Inspecting Local Sockets with `ss`

The ss (Socket Statistics) utility inspects active internal sockets on Linux platforms and serves as a modern replacement for the legacy netstat tool.

Run the diagnostic command to check socket state.

$ sudo ss -lntp

Flag Specifications:

-l: Displays only listening (LISTEN) sockets.
-n: Displays numeric addresses and ports (prevents overhead from DNS and service name resolution).
-t: Displays TCP sockets.
-p: Displays process IDs and names associated with sockets (requires root or sudo privileges).

Executing the command outputs socket information.

State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1234,fd=3))
LISTEN 0 128 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=1500,fd=6))
LISTEN 0 128 127.0.0.1:8080 0.0.0.0:* users:(("java",pid=2000,fd=20))

8. Analyzing Bind Addresses: `127.0.0.1` vs `0.0.0.0`

The Local Address:Port bind configuration dictates external reachability.

Loopback Isolation Case (`127.0.0.1:8080`)

Indicates a state where the socket is bound exclusively to the loopback address.

LISTEN 0 128 127.0.0.1:8080 0.0.0.0:*

The application process (java) is bound exclusively to the loopback interface (lo).

[ External Client ] [ Target Server ]
 192.168.10.50 192.168.10.100
 | |
 |--- HTTP request to 192.168.10.100:8080 ---------&gt;| [ Physical Interface ]
 | |
 | (Blocked / No socket)
 | v
 | [ Loopback Interface ]
 | 127.0.0.1:8080 (Java Daemon)

Local Verification: Executing curl http://127.0.0.1:8080 on the server succeeds.

Remote Verification: Executing curl http://192.168.10.100:8080 from an external client fails.

Wildcard IPv4 Bind (`0.0.0.0:8080`)

Indicates a state where the socket is bound across all interfaces.

LISTEN 0 128 0.0.0.0:8080 0.0.0.0:*

The socket accepts inbound TCP traffic through all active IPv4 network interfaces assigned to the host (e.g., ens160 with IP 192.168.10.100). If permitted by firewalls, connections from external clients are possible.

+-------------------------------------------------------------+
| Server: 192.168.10.100 |
| |
| Interface ens160 [ 192.168.10.100 ] &lt;--- [ External Request ] |
| | |
| v |
| Service bound to 0.0.0.0:8080 (Listening on all interfaces) |
+-------------------------------------------------------------+

9. Real-World Troubleshooting Cases

Case 1: Ping Succeeds, but SSH Fails (`Connection Refused`)

Environment Configuration:

Client: 192.168.10.50
Server: 192.168.10.100
Target Service: SSH (TCP port 22)

Step 1: Client-Side Diagnostics

$ ping 192.168.10.100
# Result: 64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.3 ms (Success)

$ nc -zv 192.168.10.100 22
# Result: nc: connect to 192.168.10.100 port 22 failed: Connection refused

Step 2: Server-Side Socket Inspection

$ sudo ss -lntp
# Output result:
# LISTEN 0 128 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=1500,fd=6))

TCP port 22 is absent from the listener table.

Step 3: Verifying Service State and Resolution

$ sudo systemctl status sshd
# Output: Active: inactive (dead)

$ sudo systemctl start sshd
$ sudo systemctl enable sshd

$ sudo ss -lntp | grep ':22'
# Output: LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=3102,fd=3))

Re-executing nc -zv 192.168.10.100 22 on the client side results in a successful connection.

Case 2: Process Running, but External Access Fails (`Bind Mismatch`)

Diagnostic State:

Executing ss -lntp on the application server yields the socket state.

LISTEN 0 128 127.0.0.1:8080 0.0.0.0:* users:(("node",pid=4211,fd=18))

Reachability Testing:

From Target Server: curl http://127.0.0.1:8080 $\longrightarrow$ 200 OK
From Remote Client: curl http://192.168.10.100:8080 $\longrightarrow$ Connection Refused

Diagnostics and Solution:

The application (Node.js, Spring Boot, Nginx, etc.) configuration is restricted to localhost. Modify the host configuration (e.g., server.host = 0.0.0.0 or LISTEN 0.0.0.0:8080) and restart the service to expose the socket to external interfaces.

Case 3: Service Bound, but External Connection Times Out

Diagnostic State:

The server’s socket table is properly exposed.

$ sudo ss -lntp | grep ':8080'
LISTEN 0 128 0.0.0.0:8080 0.0.0.0:* users:(("java",pid=8820,fd=7))

However, client-side execution fails.

$ nc -zv 192.168.10.100 8080
nc: connect to 192.168.10.100 port 8080 failed: Connection timed out

Firewall Inspection:

Because the output is Timeout rather than Refused, an inline network device or host firewall rule is silently dropping packets.

Inspect localhost rules.

$ sudo firewall-cmd --list-all
# Check if '8080/tcp' exists in the services and ports section

$ sudo nft list ruleset

If ICMP is globally permitted in firewalld but port 8080 is excluded, ping passes through while L4 TCP traffic to port 8080 is dropped.

Client (192.168.10.50) Server (192.168.10.100)
 | |
 |----------------------- TCP SYN ----------------------------&gt;|
 | |
 | [ Host Firewall (nftables) ]
 | [ Action: DROP Port 8080 ]
 | |
 | &lt;=== Timeout (SYN-ACK is not sent) =========================|

10. Deep Packet Diagnostics with `tcpdump`

nc reports connection results, and ss indicates socket listening state. However, tcpdump captures actual packet movement and reveals state transitions at the frame level.

Run packet capture on the target server.

$ sudo tcpdump -nn -i ens160 tcp port 8080

-n`: Do not resolve hostnames.

-nn: Do not resolve hostnames or port names (displays numeric IPs and ports).
-i ens160: Monitors traffic on target network interface ens160.
tcp port 8080: Filters strictly TCP packets passing through port 8080.

11. Packet Pattern Analysis with `tcpdump`

Scenario A: Normal TCP 3-Way Handshake

10:00:01.001 IP 192.168.10.50.52344 &gt; 192.168.10.100.8080: Flags [S], seq 1001
10:00:01.002 IP 192.168.10.100.8080 &gt; 192.168.10.50.52344: Flags [S.], seq 2001, ack 1002
10:00:01.003 IP 192.168.10.50.52344 &gt; 192.168.10.100.8080: Flags [.], seq 1002, ack 2002

Flags [S]`: Initial SYN frame from client.

Flags [S.]: SYN-ACK response from server.
Flags [.]: ACK frame from client.
Diagnosis: L4 handshake completed and connection established (ESTABLISHED).

Scenario B: Sustained Unanswered Inbound SYN

10:00:01.001 IP 192.168.10.50.52344 &gt; 192.168.10.100.8080: Flags [S], seq 1001
10:00:04.005 IP 192.168.10.50.52344 &gt; 192.168.10.100.8080: Flags [S], seq 1001
10:00:10.009 IP 192.168.10.50.52344 &gt; 192.168.10.100.8080: Flags [S], seq 1001

Observation: The client is retransmitting TCP SYN. The server’s interface receives the packet, but no outbound SYN-ACK ([S.]) is sent.

Diagnosis: A host-level security mechanism (iptables, nftables, SELinux, or kernel-level IP filtering) drops the packet immediately upon ingress at the network interface layer.

Scenario C: Server Sends SYN-ACK, but Client Retransmits SYN

10:00:01.001 IP 192.168.10.50.52344 &gt; 192.168.10.100.8080: Flags [S]
10:00:01.002 IP 192.168.10.100.8080 &gt; 192.168.10.50.52344: Flags [S.]
10:00:04.005 IP 192.168.10.50.52344 &gt; 192.168.10.100.8080: Flags [S]

Observation: The server processes inbound SYN and transmits SYN-ACK, but the client continuously sends SYN frames as if no response was received.

Diagnosis: A return path routing failure between server and client, or egress firewall filtering toward the source direction, is occurring.

Client (192.168.10.50) Server (192.168.10.100)
 | |
 |----------------------- 1. SYN -----------------------------&gt;|
 | |
 | X &lt;=== [ Return routing failure / Egress block ] ---------| (Sending SYN-ACK)

Primary causes:

Misconfigured local default gateway on the target server.
Return path drops due to asymmetric routing policies.
Loss of NAT table state.

Scenario D: Immediate Reset (`RST` Immediately Following `SYN`)

10:00:01.001 IP 192.168.10.50.52344 &gt; 192.168.10.100.8080: Flags [S]
10:00:01.001 IP 192.168.10.100.8080 &gt; 192.168.10.50.52344: Flags [R.]

Observation: The server immediately returns a reset frame ([R.]).

Diagnosis: The packet passed through the host firewall, but the kernel found no process listening on TCP port 8080. Verify daemon status with ss -lntp.

12. Tracking Socket State via Kernel Counters

Run ss -ant to review connection state allocation across active sockets.

$ sudo ss -ant
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:8080 0.0.0.0:*
ESTAB 0 0 192.168.10.100:8080 192.168.10.50:52344
SYN-RECV 0 0 192.168.10.100:8080 192.168.10.50:52345

Critical Socket States and Anomalies:

Accumulation of SYN-SENT (Client Side):
```
$ ss -ant state syn-sent
```

Meaning*: The client application transmitted a SYN frame but has not received a SYN-ACK. Indicates outbound blocking or drop along the path.

Accumulation of SYN-RECV (Server Side):
```
$ ss -ant state syn-recv
```

Meaning*: The server received SYN and transmitted SYN-ACK, but remains stalled waiting for the final ACK from the client. Indicates inbound blocking on the client side, asymmetric routing, or a SYN flood attack.

13. Upper-Layer Diagnostics: Ping and TCP Succeed, but Application Fails

Define a further elevated failure state.

$ ping 192.168.10.100
# Success (L3 OK)

$ nc -zv 192.168.10.100 8080
# Success (L4 TCP connection OK)

$ curl http://192.168.10.100:8080/
# Failure / Error / HTTP 5xx / No response

This scenario proves that the network layer (L3) and transport layer (L4) are fully operational.

Application Layer Debugging with `curl`

Execute verbose HTTP operations to capture handshake and protocol semantics.

$ curl -v http://192.168.10.100:8080/

Example of a verbose execution trace:

* Trying 192.168.10.100:8080...
* Connected to 192.168.10.100 (192.168.10.100) port 8080 (#0)
&gt; GET / HTTP/1.1
&gt; Host: 192.168.10.100:8080
&gt; User-Agent: curl/7.76.1
&gt; Accept: */*
&gt;
&lt; HTTP/1.1 504 Gateway Timeout
&lt; Server: nginx/1.20.1
&lt; Date: Mon, 10 Nov 2025 10:00:00 GMT
&lt; Content-Type: text/html

Primary Cause Categories for L7 Failures:

Reverse Proxy Upstream Timeout: Nginx accepts connections on port 8080, but fails to reach the backend running locally (e.g., Node/Java app on 127.0.0.1:8000), returning HTTP 504 Gateway Timeout or 502 Bad Gateway.
HTTP Host Header Rejection: Application virtual host filters drop requests lacking a valid Host: request header.
Application Thread Lock: Database connection pool exhaustion or internal deadlocks prevent the application layer from transmitting payload frames despite an established L4 socket handshake.
TLS/SSL Handshake Failure: Cipher suite mismatches, expired certificates, or SNI misconfigurations cause connection termination immediately following TCP initialization on port 443.

14. Caveats in UDP Protocol Diagnostics

Unlike TCP, UDP is connectionless. There are no setup flags (SYN, ACK) or state machine checks.

Testing UDP reachability using nc requires the -u flag.

$ nc -zvu 192.168.10.100 53

Diagnostic Challenges:

UDP scanners transmit 0-byte packets. If no ICMP Port Unreachable frame returns, nc determines the port to be “open” or “open|filtered”.
Due to a high rate of false positives, reliable UDP testing requires protocol-specific query tools (e.g., dig for DNS UDP 53, snmpwalk for SNMP UDP 161) alongside active packet capture.

$ sudo tcpdump -nn -i ens160 udp port 53

15. Tool Matrix for Systems Engineers

Integrate these diagnostic tools into a unified operational workflow.

Utility	Target Layer	Core Functionality	Primary Diagnostic Value
`ping`	L3 (Network)	Raw ICMP reachability and link latency evaluation	Verification of basic IP routing and physical connectivity
`nc`	L4 (Transport)	End-to-end TCP/UDP socket availability testing	Differentiation between `Connection Refused` and `Timeout`
`ss`	Host Kernel Socket	Local interface bind and socket state inspection	Confirmation that daemon processes are properly started and bound
`tcpdump`	Wire Protocol	Monitoring TCP frames on raw network interfaces	Failure location identification via sequence flag (`SYN`, `SYN-ACK`, `RST`) analysis
`curl`	L7 (Application)	HTTP/S application protocol transaction validation	Identification of web server errors, proxy misconfigurations, and backend timeouts

16. Systematic Troubleshooting Flow

🛠️ When encountering an event where “Ping succeeds, but service access fails”, proceed with diagnostics according to the following logical flow.

[ Event Occurred: Ping is successful, but service access fails ]
 |
 v
 Execute: nc -zv <server_ip> <port>
 |
 +-----------------------------------+-----------------------------------+
 | | |
 v v v
 [ Connection Refused ] [ Connection Timed Out ] [ Succeeded ]
 | | |
 v v v
 Log in to target server Run packet capture on server: L3/L4 layers are normal.
 Execute: ss -lntp tcpdump -nn -i <nic> port <port> Verify L7 application layer.
 | | Execute: curl -v http://<ip>:<port>
 +---------------+ +---------------+ |
 | | | | v
 v v v v Check app/proxy logs:
 [ Port not listed ] [ Bound to 127.0.0.1 ] [ SYN not reached ] [ SYN reached, no SYN-ACK ] - Systemd status
 | | | | - Nginx / App exceptions
 v v v v - DB connection pool
 Start daemon: Fix app config Check network path: Check host firewall:
 systemctl start 0.0.0.0:<port> - Network FW - firewall-cmd
 <service> Restart app - Security Group - nftables rules
 - Router settings - Kernel IP filtering

17. Operational CLI Quick Reference

L3 Diagnostic Commands:

# Send 4 ICMP packets to test link reachability
$ ping -c 4 192.168.10.100

L4 Socket Probing Commands (`nc`):

# Investigate standard management and web service TCP ports
$ nc -zv 192.168.10.100 22
$ nc -zv 192.168.10.100 80
$ nc -zv 192.168.10.100 443
$ nc -zv 192.168.10.100 8080

# Investigate UDP domain name resolution service
$ nc -zvu 192.168.10.100 53

Local Socket Inspection Commands (`ss`):

# List all active listening TCP ports including process details
$ sudo ss -lntp

# Filter for a specific target socket listener
$ sudo ss -lntp | grep ':8080'

# Display all numeric TCP connections regardless of state
$ sudo ss -ant

# Isolate connections in SYN-SENT or SYN-RECV state
$ sudo ss -ant state syn-sent
$ sudo ss -ant state syn-recv

Packet Analysis Commands (`tcpdump`):

# Monitor TCP port 8080 on a specific interface
$ sudo tcpdump -nn -i ens160 tcp port 8080

# Capture raw SYN connection initialization requests globally
$ sudo tcpdump -nn -i ens160 'tcp[tcpflags] &amp; tcp-syn != 0'

# Capture all traffic associated with a specific host IP address
$ sudo tcpdump -nn -i ens160 host 192.168.10.50

# Multiplex tracking of ICMP control packets and target TCP application traffic
$ sudo tcpdump -nn -i ens160 'tcp port 8080 or icmp'

Host Firewall Operations (`firewalld` / `nftables`):

# Inspect current firewalld configuration
$ sudo firewall-cmd --list-all

# Permanently add permission for TCP 8080
$ sudo firewall-cmd --add-port=8080/tcp --permanent
$ sudo firewall-cmd --reload

# Inspect low-level nftables ruleset bindings
$ sudo nft list ruleset

Lessons Learned

Ping Success Does Not Guarantee Service Health: ICMP Echo replies prove only Layer 3 interface reachability and routing functionality. Layer 4 port states and Layer 7 application functionality are not verified.
Accurate Error Message Interpretation:

Connection refused indicates that the packet reached the host, but there is no active listener, or it was explicitly rejected with a RST. Check host socket bindings (ss -lntp).
Connection timed out indicates that frames were lost on the forward or return path. Check firewalls, security groups, and routing loops.

Packet Path Tracing via Wire Capture: Use tcpdump to answer the fundamental question: “How far into the infrastructure stack did the initial SYN frame reach?”
Step-by-Step Layer Isolation: Sequentially test from L3 (ping) to L4 (nc), inspect local listening states (ss), verify raw frames on the wire (tcpdump), and finally validate at L7 (curl).</server_ip>

Nc on K-Life Hack | Systems Architecture & DevOps

Troubleshooting the Boundary Area Where Ping Succeeds but TCP Connection Fails

1. Network Layer Abstraction and Service Mapping

2. Low-Layer Mechanics of ping

3. Structure of the TCP 3-Way Handshake

4. Diagnosing “Ping Succeeds + TCP Connection Fails”

5. Transport Layer Probing with nc (Netcat)

6. Mechanisms of Connection Refused and Connection Timeout

Mechanics of Connection Refused

Primary Causes:

Mechanics of Connection Timeout

Primary Causes:

7. Inspecting Local Sockets with ss

Flag Specifications:

8. Analyzing Bind Addresses: 127.0.0.1 vs 0.0.0.0

Loopback Isolation Case (127.0.0.1:8080)

Wildcard IPv4 Bind (0.0.0.0:8080)

9. Real-World Troubleshooting Cases

Case 1: Ping Succeeds, but SSH Fails (Connection Refused)

Environment Configuration:

Step 1: Client-Side Diagnostics

Step 2: Server-Side Socket Inspection

Step 3: Verifying Service State and Resolution

Case 2: Process Running, but External Access Fails (Bind Mismatch)

Diagnostic State:

Reachability Testing:

Diagnostics and Solution:

Case 3: Service Bound, but External Connection Times Out

Diagnostic State:

Firewall Inspection:

10. Deep Packet Diagnostics with tcpdump

11. Packet Pattern Analysis with tcpdump

Scenario A: Normal TCP 3-Way Handshake

Scenario B: Sustained Unanswered Inbound SYN

Scenario C: Server Sends SYN-ACK, but Client Retransmits SYN

Scenario D: Immediate Reset (RST Immediately Following SYN)

12. Tracking Socket State via Kernel Counters

Critical Socket States and Anomalies:

13. Upper-Layer Diagnostics: Ping and TCP Succeed, but Application Fails

Application Layer Debugging with curl

Primary Cause Categories for L7 Failures:

14. Caveats in UDP Protocol Diagnostics

Diagnostic Challenges:

15. Tool Matrix for Systems Engineers

16. Systematic Troubleshooting Flow

17. Operational CLI Quick Reference

L3 Diagnostic Commands:

L4 Socket Probing Commands (nc):

Local Socket Inspection Commands (ss):

Packet Analysis Commands (tcpdump):

Host Firewall Operations (firewalld / nftables):

Lessons Learned

2. Low-Layer Mechanics of `ping`

5. Transport Layer Probing with `nc` (Netcat)

7. Inspecting Local Sockets with `ss`

8. Analyzing Bind Addresses: `127.0.0.1` vs `0.0.0.0`

Loopback Isolation Case (`127.0.0.1:8080`)

Wildcard IPv4 Bind (`0.0.0.0:8080`)

Case 1: Ping Succeeds, but SSH Fails (`Connection Refused`)

Case 2: Process Running, but External Access Fails (`Bind Mismatch`)

10. Deep Packet Diagnostics with `tcpdump`

11. Packet Pattern Analysis with `tcpdump`

Scenario D: Immediate Reset (`RST` Immediately Following `SYN`)

Application Layer Debugging with `curl`

L4 Socket Probing Commands (`nc`):

Local Socket Inspection Commands (`ss`):

Packet Analysis Commands (`tcpdump`):

Host Firewall Operations (`firewalld` / `nftables`):