Squid Proxy server has stopped handling connection, resulting in NONE/503 0 CONNECT errors

A NONE/503 error in Squid Proxy typically indicates a service unavailable error. When it occurs for a specific proxy connection, it suggests an issue with that particular destination or an issue in the network communication. Here’s a step-by-step guide to troubleshoot and resolve this issue:

1. Check Squid Logs

Examine Squid’s access log and cache log for detailed error messages:

• Access Log: This log records client requests and server responses. Look for entries related to the problematic connection.
• Cache Log: This log includes detailed error messages and can provide more context about why the connection failed.

$ tail -f /var/log/squid/access.log
$ tail -f /var/log/squid/cache.log

2. Verify Destination Server Availability

Ensure the destination server is up and reachable. You can do this by:

• Ping/curl the Server: Check network connectivity to the server.

$ curl -v telnet://destination_server_ip_or_hostname:port 
$ nc -zv destination_server_ip_or_hostname port
$ telnet destination_server_ip_or_hostname port

Optional: By taking a packet capture on the squid server using 
            tcpdump/wireshark the issue can be esily identified. 

3. DNS Resolution

Ensure that the system on which Squid is installed can resolve the destination server’s hostname correctly:

• Test DNS Resolution: Use nslookup or dig to verify DNS resolution

$ nslookup destination_server_hostname
$ dig destination_server_hostname

4. Firewall and Network Configuration

Ensure there are no firewall rules(local/enterprise) blocking the connection.

5. Squid Configuration

Verify Squid configuration for any misconfigurations:

• acl and http_access: Ensure there are no ACLs (Access Control Lists) or http_access rules blocking the connection in the /etc/squid/squid.conf configuration file.
• Cache Peer: If using cache peers, verify their configuration and ensure they are accessible.

cache_peer hostname parent port 0 no-query no-digest

6. Test with Direct Connection

Try accessing the destination server directly from the Squid server to rule out Squid-specific issues.

• Curl or Wget: Use curl or wget to make a direct connection.

$ curl -v https://destination_server_hostname:port
$ curl -v telnet://destination_server_hostname:port
$ wget https://destination_server_hostname:port

7. Packet Trace using tcpdump or wireshark

The issue can be easily identified by capturing packets on the Squid server using tools like tcpdump or Wireshark.

$ tcpdump -i <eth-iface-name> -s0 -w /tmp/squid.pcap

How I resolved the issue?

• I found that squid was logging the following events in its access log file.

x.x.x.x NONE/503 0 CONNECT test.example.local:443 - HIER_NONE/- -
x.x.x.x NONE/503 0 CONNECT test.example.local:443 - HIER_NONE/- -
x.x.x.x NONE/503 0 CONNECT test.example.local:443 - HIER_NONE/- -

• I took a network packet capture on the squid server using tcpdump command.

$ tcpdump -i <eth-iface-name> -s0 -w /tmp/squid.pcap

• I found that Squid first queried the DNS server to resolve the domain test.example.local for both IPv4 and IPv6 addresses. The DNS server correctly responded with the A (IPv4) and AAAA (IPv6) records for test.example.local.

$ tshark -r squid.pcap -Y "dns.qry.name==test.example.local"
 1  18.661127  x.x.x.1 → x.x.x.2  DNS 91 Standard query 0xb118 A test.example.local
 2  18.661159  x.x.x.1 → x.x.x.2  DNS 91 Standard query 0x4ab9 AAAA test.example.local
 3  18.661488  x.x.x.2 → x.x.x.1  DNS 248 Standard query response 0xb118 A test.example.local A y.y.y.1 A y.y.y.2 A y.y.y.3 
 4  18.666566  x.x.x.2 → x.x.x.1  DNS 315 Standard query response 0x4ab9 AAAA test.example.local AAAA 2620:y:y:1 AAAA 2620:y:y:2 AAAA 2620:y:y:3

• However, Squid chose to first communicate with the IPv6 address of test.example.local, which failed because the network environment does not support IPv6.
• In the following example, TCP SYN packet to the IPV6 address 2620:y:y:1(test.example.local) is getting retransmitted because the there is no reply from the destination and this is because the network environment doesn’t support IPV6 networking.

$ tshark -r squid.pcap -Y "ipv6.addr == 2620:y:y:1 or ipv6.addr==2620:y:y:2 or ipv6.addr==2620:y:y:3 or ip.addr==y.y.y.1 or ip.addr==y.y.y.2 or ip.addr==y.y.y.3"

 1  16.195064 fe80::abcdxxx:yyy:zzz → 2620:y:y:1 TCP 96 49490 → 443 [SYN] Seq=0 Win=28800 Len=0 MSS=1440 SACK_PERM TSval=3944462120 TSecr=0 WS=128
 2  17.258090 fe80::abcdxxx:yyy:zzz → 2620:y:y:1 TCP 96 [TCP Retransmission] 49490 → 443 [SYN] Seq=0 Win=28800 Len=0 MSS=1440 SACK_PERM TSval=3944463183 TSecr=0 WS=128
 3  19.258113 fe80::abcdxxx:yyy:zzz → 2620:y:y:1 TCP 96 [TCP Retransmission] 49490 → 443 [SYN] Seq=0 Win=28800 Len=0 MSS=1440 SACK_PERM TSval=3944463183 TSecr=0 WS=128
 4  31.258113 fe80::abcdxxx:yyy:zzz → 2620:y:y:1 TCP 96 [TCP Retransmission] 49490 → 443 [SYN] Seq=0 Win=28800 Len=0 MSS=1440 SACK_PERM TSval=3944463183 TSecr=0 WS=128

As per squid official doc, with the IPv6 Internet being as fast or faster than IPv4 Internet for dual-stack domains Squid prefers to contact websites over IPv6 address. Since the OS supports IPV6 networking even though the underlying networking environment doesn’t, squid explicitly chose to communicate with IVP6 address of test.example.local.

Such a behavior can be reversed by adding the following settings(between ACL and http_access allow section) in the squid.conf file and then restart the squid proxy service.

dns_v4_first on

The above configuration option reverses the order of preference nad forces Squid to contact dual-stack websites over IPv4 first. Squid will still perform both IPv6 and IPv4 DNS lookups before connecting.

Alternatively, disabling IPV6 networking on the Squid server will force SQUID to use IPV4 networking only.

# disable for all interfaces by adding the following line in /etc/sysctl.conf 
net.ipv6.conf.all.disable_ipv6 = 1