That shouldn't be happening, as masquerading is (or should be) done after the output ISP is chosen.

Has anyone observed this issue, can anyone suggest the solution ?

Regarding the only internal network, I hope the changes are obvious (just remove everything where eth1 is referenced, as you only have eth0).

Since you don't want load balancing but only failover, this makes things much easier as you don't need all the fancy marking stuff, nor multiple routing tables. Also remove the parts that deal with the iptables' statistic match and just point the default route to the ISP you want to be the primary one.

Now, when you detect failover you just point the default route to the "other" ISP.

Regarding IPv6 NAT (which I personally find unnecessary), you'll have to do pretty much the same thing that is done for IPv4. Sample failover script follows:

#!/bin/bash
 
# using associative arrays to store the information
 
declare -a isp
declare -A iface ip ip6 localip localip6 status
 
isp=( ISP1 ISP2 )
 
iface["ISP1"]="eth2"
iface["ISP2"]="eth3"
 
ip["ISP1"]="1.1.1.2"
ip["ISP2"]="2.2.2.2"
 
ip6["ISP1"]="2001:db8:0:1::2"
ip6["ISP2"]="2001:db8:0:2::2"

localip["ISP1"]="1.1.1.1"
localip["ISP2"]="2.2.2.1"

localip6["ISP1"]="2001:db8:0:1::1"
localip6["ISP2"]="2001:db8:0:2::2"

statedir=/var/tmp

# check which ISP is up
 
for i in "${isp[@]}"; do
  # if there's no state file for the ISP, assume it's up
  if [ -f "${statedir}/${i}_state" ]; then
    status[$i]=$(< "${statedir}/${i}_state")
  else
    status[$i]="up"
  fi
done

# if ISP1 is up, use it

if [ "${status[ISP1]}" = 'up' ]; then

  outiface=${iface[ISP1]}
  outip=${ip[ISP1]}
  outlocalip=${localip[ISP1]}
  outip6=${ip6[ISP1]}
  outlocalip6=${localip6[ISP1]}

elif [ "${status[ISP2]}" = 'up' ]; then

  outiface=${iface[ISP2]}
  outip=${ip[ISP2]}
  outlocalip=${localip[ISP2]}
  outip6=${ip6[ISP2]}
  outlocalip6=${localip6[ISP2]}

else
  # no ISP is up, exit
  echo "No ISP is up, terminating!" >&2
  exit 1
fi
 
# IPv4
 
# flush everything
iptables -F
iptables -t nat -F
 
# do NAT first

# SNAT packets going out; MASQ may also be used instead
iptables -t nat -A POSTROUTING -o "$outiface" -j SNAT --to-source "$outlocalip"
 
# routing
 
ip route del default
ip route add default dev "${outiface}" via "${outip}"
ip route flush cache


# IPv6
 
# flush all the rules
ip6tables -F
ip6tables -t nat -F
 
# do NAT first

# SNAT packets going out; MASQ may also be used instead
ip6tables -t nat -A POSTROUTING -o "$outiface" -j SNAT --to-source "$outlocalip6"
 
# routing

ip -6 route del default
ip -6 route add default dev "${outiface}" via "${outip6}"
ip -6 route flush cache

WARNING: this is untested, but should give you a starting point. Of course you have to do the necessary adjustments like adding custom iptables rules which you surely have, and replace the IP addresses and interface names with your actual ones.