Difference between revisions of "Troubleshooting"

From securityrouter.org, an OpenBSD-based firewall
Jump to: navigation, search
 
(5 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
This page contains a collection of different troubleshooting techniques that can be used in different scenarios.
 
This page contains a collection of different troubleshooting techniques that can be used in different scenarios.
  
== Common troubleshooting tasks ==
+
== Routing and firewalling ==
Since the security router is a network product, most troubleshooting articles goes into this section.
 
 
 
=== Routing and firewalling ===
 
 
Troubleshooting routing and firewalling issues are closely related, and therefore covered in the same section.
 
Troubleshooting routing and firewalling issues are closely related, and therefore covered in the same section.
  
==== You've added a rule or port forward, but it doesn't work ====
+
=== Tracing packets to debug firewall or routing issues ===
 
It could be either a firewall or routing problem. Start out by seeing if the packet is blocked by running
 
It could be either a firewall or routing problem. Start out by seeing if the packet is blocked by running
 
  firewall log host some-ip-address
 
  firewall log host some-ip-address
 
where you filter the log based on the source IP address of the connection you're trying to get through. You can also use filters such as "port XXX" is that's more suitable. If nothing shows up, try showing ''all traffic'' (shows everything, either allowed or blocked) on the interface on which the connection is supposed to enter the router
 
where you filter the log based on the source IP address of the connection you're trying to get through. You can also use filters such as "port XXX" is that's more suitable. If nothing shows up, try showing ''all traffic'' (shows everything, either allowed or blocked) on the interface on which the connection is supposed to enter the router
 
  tcpdump -n -i em0 host some-ip-address
 
  tcpdump -n -i em0 host some-ip-address
where ''em0'' represents the WAN interface, which is logical in case we're debugging a port forwarding where the test connection is coming form the Internet. In case you see the connection, move on to the next interface where you believe the connection should exit the router
+
where ''em0'' represents the WAN interface, which is logical in case we're debugging a port forwarding where the test connection is coming form the Internet. In case you see the connection, move on to the next interface where you believe the connection should exit the router. In other word, execute the same command again, but replace ''em0'' with for example ''em1'' (might be another interface in your case). If you don't see the connection leaving the firewall, examine the routing table and your firewall rules. If you do in fact see traffic leaving, but not returning from the server (or other device you are trying to connect to), start troubleshooting that device to see why the traffic is not returned.
tcpdump -n -i em1 host some-ip-address
 
where ''em1'' represents the LAN interface (might be another interface in your case). If you don't see the connection leaving the firewall, examine the routing table and your firewall rules. If you do in fact see traffic leaving, but not returning from the server (or other device you are trying to connect to), start troubleshooting that device to see why the traffic is not returned.
 
  
==== You've added rules you believe should work, but they don't ====
+
=== Firewall rules that doesn't seem to apply ===
 
The firewall's state table is evaluated before the rules, and an existing state entry might prevent your rule from "winning". By for example restarting a continuous ping, the old state is no longer matches. The other method is to simply [[Firewalling#Flushing_states|flush the states]].
 
The firewall's state table is evaluated before the rules, and an existing state entry might prevent your rule from "winning". By for example restarting a continuous ping, the old state is no longer matches. The other method is to simply [[Firewalling#Flushing_states|flush the states]].
  
=== Logging ===
+
== System ==
This section in concerned with system logs. For traffic dumps or firewall logs, see [[#Routing and firewalling|that section]].
+
Anything related to the system or device itself.  
 
 
==== The system log doesn't show  ====
 
Normally the logs are stored in RAM memory. However, if a storage device is used for logging, and that device somehow becomes broken or inaccessible, you can still view the kernel's in-memory log by running <tt>dmesg</tt>.
 
  
=== System and hardware ===
+
=== Logging issues  ===
Anything else related to the system or device itself.
+
Normally the logs are stored in RAM memory. However, if a storage device is used for logging, and that device somehow becomes broken or inaccessible, you can still view the kernel's in-memory log by running <tt>dmesg</tt>. The most common cause of this problem is if unplugging a storage disk while running, perhaps after performing a storage-based software update. Only unplug disks when the system is shut down.  
  
==== Rebooting doesn't work ====
+
=== Rebooting issues ===
In case justing pressing the Reboot button, or the <tt>reboot</tt> [[CLI]] command doesn't work, the [[backend]] process might have crashed. Try running <tt>reboot -f</tt> instead. If that doesn't work either, enable/use the [[root access]] to run <tt>reboot -n -q</tt>.
+
In case rebooting doesn't work, the [[backend]] process might have crashed. Try running <tt>reboot -f</tt> instead. If that doesn't work either, enable/use the [[root access]] to run <tt>reboot -n -q</tt> to perform a quick and dirty system reboot.

Latest revision as of 10:21, 22 February 2013

This page contains a collection of different troubleshooting techniques that can be used in different scenarios.

Routing and firewalling

Troubleshooting routing and firewalling issues are closely related, and therefore covered in the same section.

Tracing packets to debug firewall or routing issues

It could be either a firewall or routing problem. Start out by seeing if the packet is blocked by running

firewall log host some-ip-address

where you filter the log based on the source IP address of the connection you're trying to get through. You can also use filters such as "port XXX" is that's more suitable. If nothing shows up, try showing all traffic (shows everything, either allowed or blocked) on the interface on which the connection is supposed to enter the router

tcpdump -n -i em0 host some-ip-address

where em0 represents the WAN interface, which is logical in case we're debugging a port forwarding where the test connection is coming form the Internet. In case you see the connection, move on to the next interface where you believe the connection should exit the router. In other word, execute the same command again, but replace em0 with for example em1 (might be another interface in your case). If you don't see the connection leaving the firewall, examine the routing table and your firewall rules. If you do in fact see traffic leaving, but not returning from the server (or other device you are trying to connect to), start troubleshooting that device to see why the traffic is not returned.

Firewall rules that doesn't seem to apply

The firewall's state table is evaluated before the rules, and an existing state entry might prevent your rule from "winning". By for example restarting a continuous ping, the old state is no longer matches. The other method is to simply flush the states.

System

Anything related to the system or device itself.

Logging issues

Normally the logs are stored in RAM memory. However, if a storage device is used for logging, and that device somehow becomes broken or inaccessible, you can still view the kernel's in-memory log by running dmesg. The most common cause of this problem is if unplugging a storage disk while running, perhaps after performing a storage-based software update. Only unplug disks when the system is shut down.

Rebooting issues

In case rebooting doesn't work, the backend process might have crashed. Try running reboot -f instead. If that doesn't work either, enable/use the root access to run reboot -n -q to perform a quick and dirty system reboot.