Friday, 21 December 2012

pfSense bridge gateway vmware ovh ip failover ripe

This post is under construction.

This post is about the setup of a pfSense 2.0 firewall in transparent mode to connect and protect virtual machines hosted by OVH to the internet.

OVH provides failover IPs (fr) and RIPE IP Failover block (fr). These IPs and blocks can be re-routed between servers you are renting inside an OVH datacenter in 1min.
This make migration, failover and HA easy and at a low cost, no need to change something in the DNS or in the VM setup.

The setup and the use of these IPs is not easy. Adding a firewall in between don't help.To not increase the complexity of this post, the firewall is configured in transparent mode. This means that interfaces of the firewall are bridged.

What is a bridge ?

A network bridge fits between two networks and repeats Ethernet frames on one side to the other side and vice versa. A bridge works at layer 2 (Ethernet) of the OSI model (IP is layer 3) and ignores the IP settings. All type of Ethernet frames are forwarded by the bridge, even theses that encapsulate packets that are not related to the IP protocol. Bridges are smart and learn MAC addresses on both side to avoid to repeat frames that don't need to cross from one network to the other.

Promiscuous mode

If you are using an hypervisor like VMware ESXi or Xen, you have to enable the promiscuous mode on each interfaces of the bridge to allow it to see all Ethernet frames and guess host on each side.

Here is the procedure for VMware, ask Google if you use another hypervisor.


pfSense and bridging

The bridge connects multiple interfaces. Each of these interfaces can have an IP address, but this is not required. The Bridge itself can be attached to a network interface and have an IP address, but this is not required either. Here the goal is not to combine two networks in one but just to connect them without wasting any IP addresses or increasing complexity. Frames that contains IP packets can be filtered by the firewall rules. Here I'll bridge the WAN and the OPT1 (DMZ) interface. The LAN will be kept out of the bridge.
pfSence can filter at the bridge level, but we want to filter at the member level. Check in the advanced settings of your FW that your setup match this one:



When do you need to allocate an OVH virtual MAC for your VM

I think the main purpose of the virtual MAC at OVH is to avoid MAC address conflict. This could be very annoying if your production server would stop working because someone is installing a new VM using a random MAC address that match your ! In the mean time they can use this to reinforce the security of their network.
Most of the time when you attache a failover IP to a virtual machine then you need to allocate a virtual MAC for this VM. But his is not always true. For example if you are attaching a second IP to an existing VM using the IP Alias approach, you don't need one because this second IP will use the MAC address of the primary address. Also If your VM is behind a router doing some Proxy ARP, the router use its own MAC address to masquerade your IP and the other side of the router it will be happy with any MAC address used by your VM.
This could be counter productive to allocate a MAC address when not needed, the OVH infrastructure could block the traffic when the MAC don't match the IP address. And if their are not doing that today they could do it later.

In the configuration bellow, you need to allocate a virtual MAC address for VMs going in the DMZ and for the firewall itself if you give it an IP address. No need of virtual MAC for VMs going in the LAN.

Giving a routable IP address to the firewall or not ?

Even if the bridge mode allows the firewall to do its job without giving it any IP address, the firewall need one to be managed using a web browser.

For the early setup and for emergency access, I have the use to create a VM using a XUbuntu LiveCD distibution on the LAN side and use the VMware console to manage the firewall. The Firewall LAN side is configured with default address 192.168.1.1 and a working DHCP server, this make the network setup of the VM very easy. Just connect your browser to http://192.168.1.1, use admin / pfsense for login / password and you are in.

Giving a routable IP to the WAN side of your firewall would allow you to remotely manage your firewall and do hide NAT (masquerading) for the VMs connected to the LAN side (if any). This also allows you to upgrade and install new packages.

The challenge

Here is the "virtual" challenge I'll try to solve in this post. I'm using addresses from the private ranges to make my live easier for the demo but address in 10.x.x.x and 172.x.x.x must be viewed as routable address. 192.168.x.x are used for private network, like the LAN.

I have an OVH dedicated server with address 10.99.99.99/254 and its gateway at 10.99.99.254. OVH gives me a RIPE block of 8 addresses: 10.10.10.8/29 with the gateway at 10.10.10.14. I also have 2 single failover IPs 172.16.16.16/32 and 172.17.17.17/32.

The RIPE block can be used in 2 different ways.
  • The normal way is to use it as a subnet, with first address of the block reserved for the network and the last one for the broadcast address. Including the gateway address that is already allocated to an OVH router, this make 3 addresses unusable.  If you have a /30 block (4 addresses) this make only one usable address for your host(s).  The advantage of this setup is the simplicity of the network configuration of the hosts themselves.
  • The other possibility is to use each address individually with a netmask of  /32. You don't need to split your RIPE block in the OVH manager to works that way (this is what OVH say), just use each address like a single failover IP. When doing this, even if VMs are on the same switch, they are not on the same subnet anymore and traffic between them has to travel up to the default gateway and back. If you want a more direct path, you have to create a route between your VMs (see below). Also I thing this is better to use the RIPE gateway 10.10.10.10.14 than using the gateway of the physical host (10.99.99.254)  because when moving the VMs and the RIPE block to another host, you don't need to change the VMs gateway. The inconvenient of this choice is that you have to configure your host network using the OVH guide (french).

Simple solution for the RIPE block

In this simple setup, I don't give any routable address to the firewall. The early setup must be done with the help of a VM installed on the LAN side.


First we need to enable and setup the WAN and DMZ (OPT1) interfaces to none. We will not attach any IP addresses to these interfaces (this time). We will bridge them. Bridge works one level below the Internet protocol and don't need any IP settings.




Then we are creating the bridge :


And setup the two members of the bridge using ctrl-click. This is the most difficult part :-)


The setup is done. Yes don't be afraid this is enough. Now we have to setup the filter rules. I give you some sample rules, but use your owns.

First I like to be able to ping in any direction, at least for the setup and troubleshooting.


Here are the details :


I want to give unrestricted internet access to VMs in the DMZ. Anyway I'm adding a restriction for the LAN. The DMZ can be accessed by the internet (demilitarized zone) and can be compromised. This is why the DMZ cannot have more access rights to access the LAN than the Internet.


Here are all the details :



Now I suppose I want to host a mail server on 10.10.10.9 and need to let SMTP packets go in..



Here are the details of the SMTP setup.



Now the most important part is the network setup of 10.10.10.9. Nothing difficult  anyway :
IP_ADDR=10.10.10.9
NETMASK=255.255.255.248 (/29)
GATEWAY=10.10.10.14
Network setup for other hosts of the RIPE block are similar.

Add a single failover IP to the DMZ

Now I want to add a host with a single failover IP 172.16.16.16/32 to the DMZ. I just need to setup the filter rules like the SMTP rule above for this new host. Nothing more is required because we are working in bridge mode.
On the other hand, the setup of the host itself is more tricky, not because of the firewall, but because of how failover IP works. Follow the OVH guide (french) to setup your new host using 172.16.16.16 as the failover IP and 10.99.99.254 for the gateway (the one ending in .254).

In this configuration, traffics between 10.10.10.9 and 172.16.16.16 has to go up to the OVH router and then back to the destination. In both case the traffic has to pass through the firewall that must have a rules to let the traffic goes out and in. This is because hosts ignore they are side by side. If you expect to have a lot of traffics and want to speed it up, you have to add routes between the hosts themselves. Don't think that the fact that your traffics between host inside the DMZ goes trough the firewall increase the security. If a pirate is able to hack one of your VM inside the DMZ, he is able to bypass the firewall to attack your other hosts too.

Here are sample command lines for linux and windows. You have to hard-code these routes in the hosts configurations.

On 172.16.16.16, add a route to network 10.10.10.8/29
linux: route add -net 10.10.10.8 netmask 255.255.255.248 dev eth0
windows: route -p ADD 10.10.10.8 MASK 255.255.255.248 172.16.16.16

On 10.10.10.9 add a route to host 172.16.16.16/32
linux: route add -host 172.16.16.16 netmask 255.255.255.255 dev eth0
windows: route -p ADD 172.16.16.16 MASK 255.255.255.255 10.10.10.9

Attach an IP to the WAN side

This makes the firewall remotely manageable and allows to give an internet access to the LAN side.
I choose 10.10.10.13/29 from the RIPE block, in the next section I explain how to use a single failover IP instead.


I go to the WAN interface configuration to setup the IP and its gateway at once.


Now I masquerade the LAN addresses using hide NAT to give Internet access to the LAN side. Don't forget to switch to manual, I don't want to NAT hosts in the DMZ.



The details:


 I also need to check that the LAN has a rule that give access to the internet :


Done !

If you want, you can forward some ports to hosts inside the LAN.
You can reach host in the DMZ from LAN.

Attach a single failover IP to the WAN side

If you don't have a RIPE block or prefer to use a single failover IP instead, then read this section.
This setup cannot be configured easily using the pfSense GUI interface, some tricks are required.
Here is the plan, first get an Internet access using some hand written command lines. Then download and install the Shellcmd package. Finally hard code the command line to make the setup persistent.

First setup the WAN IP without any gateway :



Then setup the default gateway manually. I choose to use the Command Prompt interface of the GUI. Use the two commands :
route add -net 10.99.99.254/32 -iface em0
route add default 10.99.99.254


If you don't know where 10.99.99.254 is coming from, re-read the beginning of this article.
Don't forget to setup a working DNS to have a fully working internet connection. I choose the Google public DNS.


Now the firewall should have internet access and we can install package Shellcmd.


Then hard code the two commands above inside Shellcmd to get the make config persistent.


You can try to reboot, just to see if the config works and setup hide NAT for the LAN side as explained above.

Funny things 

It looks like the OVH gateway works like an universal ARP proxy or Captive portal. I means my OVH gateway replies to any ARP request. This means that for any given a.b.c.d/32 failover IP, I can setup a host or virtual host using a mask /24 (instead of /32) and a.b.c.X where X can be anything not in ( 0, 255 or d ) for my gateway.And it works on my Kimsufi server.
I think this is how OVH setup some (maybe all) of their routers to be able to support migration of  failover IP or block without too much headache.
What is fun is to traceroute some IPs in a.b.c.*  and see that they are not directly attached to the WAN but are behind some routers.
OVH don't say anything about this setup and then this feature is funny but cannot be used on a production server.
Be careful if you use a wrong setup and generate a lot of unexpected ARP requests, OVH monitor ARP request and can warn you to quickly fix the problem or even disable your network link.

Conclusion

For me the use of the bridge is a brilliant idea. The setup is simple. You can add hosts , IPs and RIPE blocks easily to the DMZ.

39 comments:

  1. No matter what I try I can not get a client (virtualized WinXP as a test) to access the internet using "Attach a single failover IP to the WAN side" method.

    My client (on 192.168.1.0/24 LAN) can ping the LAN interface and the WAN interface, and can even resolve FQDN (DNS is pointed at pFsense) but can not make any connections.

    A tracert fails after the first hop (192.168.1.1 LAN interface on pFsense)

    From pfSense itself I'm able to ping and access the internet without issue. I have shellcmd installed, and the rules seemingly hold after a reboot. I'm not sure if it's my NAT rules...

    Firewall LAN rule is enabled.

    ReplyDelete
  2. This is probably a NAT problem. You must have the same NAT setup and filter rules as in "Attach an IP to the WAN side".

    ReplyDelete
    Replies
    1. I tried that originally (or so I thought?). I went back and re-applied the NAT settings, w/o a configuration change and it worked! Perhaps it wasn't saving/applying, or more likely... user error.

      Thanks for this write-up!

      Delete
  3. Hello,
    I have configured my infrastructure like your How-To, everything is fine but i've a little problem : When I'm Connected from VPN (or LAN), I can't ping or access to a VM in DMZ (with a FailOver IP).
    Note : Is Ok From WAN Adress, this problem occured only with LAN or VPN.
    The traceroute (from VPN 172.16 Subnet):
    1 26 ms 26 ms 28 ms 172.16.0.1
    2 27 ms * * vss-1b-6k.fr.eu [94.23.38.252]
    3 * * * Délai d'attente de la demande dépassé.
    4 * * * Délai d'attente de la demande dépassé.

    The connexion stop at vss-1b-6k.fr.eu (or maybe at the pfsense firewall ??)

    Can you Help ?
    Thanks

    ReplyDelete
  4. Which VPN are you using ?
    What do you mean by "this problem occurred only with LAN " which LAN ? The LAN side of your FW or LAN at home or the office ? What is not working ?

    ReplyDelete
    Replies
    1. Hello, Thanks for the reply.
      I Use OpenVPN.
      I've 4 network :
      WAN : with my Public FailOver IP
      LAN : 10.0.0.0/16 for VMs with NAT (No FailOver on It)
      DMZ : Only for VM With IP FailOver, bridged on WAN
      OpenVPN : 172.16.0.0/24 for my VPN clients.

      From the a VM on the LAN (10.0.0.0) or from VPN Client (172.16.0.0), I can Ping and access what I want in Internet but When I want ping or access an IP in the DMZ, this error occured. (with the traceroute of my previous message).
      Sorry for the english, I'm French :)

      Thanks for your support.

      Delete
    2. Did you find a solution for this?
      I'm having the same problem: traffic from LAN is succesfullt NATted and external websites are accessible. Traffic from LAN to DMZ/OPT is not forwarded somehow. The firewall shows no info whatsoever. Mindboggling!

      Delete
  5. The LAN should have access to the DMZ.
    You should first fix this problem before to think about the VPN.
    Did'you setup the "ping in any direction" rule ? Do you have restrictions on the LAN side? To the internet or the DMZ ?

    From the VPN, do you have access to the VMs in the LAN ?

    ReplyDelete
    Replies
    1. Hi Alain,
      I'm faced the same problem as
      "Mathieu GILLOOTS" and "luca scarano30"
      can't reach dmz from my lan network.
      folowed exactly by your instructions
      could you advice anything please ?
      thank you

      Delete
  6. Yes I have the "ping in any direction"
    I does'nt have any restriction, just a fresh config like your HowTo. (+ My rules for HTTP, SMTP, ... servers).
    Lan to Internet : OK
    Lan <=> VPN : OK
    VPN to internet : OK
    DMZ <=> Internet : OK
    VPN or LAN <=> DMZ : No OK
    Like I see in the traceroute, the packets out to the OVH Router then they could'nt IN my pfsense.

    ReplyDelete
  7. I'm planning to implement your solution at Online (Dedibox) with IP Failover.
    But in the Online docs, they wrote :
    "Important : Ne jamais faire de bridge sur votre carte réseau Dedibox. Vous risqueriez de diffuser sur le réseau une adresse MAC virtuelle, ce qui provoquerait la mise en sécurité de votre port réseau et la suspension de votre serveur. Faites uniquement du NAT ou du mode routé !"
    What do you think about it ?

    ReplyDelete
    Replies
    1. If dedibox don't give you any virtual MAC address that they will not "blacklist" you cannot run a virtual firewall in bridge mode.

      Delete
    2. OK. I generated virtual mac addresses for all my failover ips in the dedibox console. So i have to associate each one to the corresponding virtual interface ?

      Delete
  8. If dedibox provides virtual MAC addresses, then give them a try.
    Be carefull, if you chnage the mac address of already installed VMs , the OS will recognize the Network adapter as a new one and will probably not reuse the configured of the "old" one.

    ReplyDelete
    Replies
    1. All right. I will try this. In the worse case my new servers will be blocked for a while but my old one will do the job.
      I will give you some feed back.

      Delete
    2. Thank you for this post. I deployed my new server with pfSense and one VM in the DMZ.
      As Dedibox provides virtual MAC adresse for my VM, all work well !

      Delete
  9. Wow, after spending half the night on it I managed to install an OVH server with Esxi, a vm inside with pfsense using a failover IP, and an other vm behind (web server) -this one now can see the internet at last.

    Now I'll install Pound on pfsense to use it as a proxy to reach the web server, but I feel it's correctly protected - being hidden from the Net.

    I just used one trick you don't use - it avoids installing a PC with a browser on the LAN leg to manage pfsense: at beginning,

    - start the pfsense vm

    - connect using the console from vsphere client

    - install the 2 routes - adding the MAC address:
    route add -net 10.99.99.254/32 -iface em0:abceef0102 (remove dots)
    route add default 10.99.99.254

    - then setup a rule to access Webconfigurator from my home/admin PC:

    easyrule pass wan tcp my.hom.e.ip fai.lov.er.ip 443

    So I can connect and go on with the install

    ReplyDelete
  10. I forgot the most important part : MERCI !

    ReplyDelete
  11. My scenario is similar:

    router nat to external esxi
    v
    v
    external linux eth with iptables (virtual machine)
    this is connected to a vshpere switch
    v
    v
    internal linux eth connected to a second vshpere switch vlan 100
    promiscuous mode as you mentioned in the tutorial
    v
    v
    in the same switch connected WAN pfsense vlan 100 promiscuos mode
    v
    v
    in another vshpere switch connected LAN pfsense
    v
    v
    in this last switch connected all the DMZ

    Not working :-(

    ReplyDelete
  12. Thanks so much.

    Worked brilliantly.

    ReplyDelete
  13. Hello,
    i've the same problem of Mathieu GILLOOTS.

    My configuration is:
    - DMZ host with failover ip and mac set from ovh, connected to dmz switch
    - DMZ interface of the firewall without ip, bridged with external eth
    - external eth without ip and connected to the public switch
    - LAN and VPN with private ip
    - WAN dedicated interface for the firewall (also gateway for the lan)

    LAN/VPN --> internet = ok
    WAN --> internet = ok
    DMZ host --> internet = ok
    WAN --> DMZ host = ok
    internet --> WAN = ok
    internet --> DMZ = ok
    LAN --> DMZ = NOT WORK

    So, all work but no the lan to dmz host connection.
    The rules are set permissive from lan but i don't see any packet filtered in dmz host's interface.
    Thank you

    ReplyDelete
  14. Hi,

    Great tutorial. The theory is usefull and working.
    But besides that PfSense is not doing the job for me.

    When everything is configured PfSense (both i386 andamd64) will become unavailable after some time. It sucks up all available CPU on my vSphere host.
    If I don't limit it, it will affect other VM's.

    When I check further... I see 100% CPU usage used by interface's taskq process according to the system activity monitor in the webinterface.

    PfSense will become unavailable.
    I have not configured any heavy traffic-using VM's on the DMZ nor LAN so I do not know what is causing this problem.

    Other problem was OVH flooding me about arp who-has scans when using the gateway of the ripe-block.
    Fixed this with ifconfig em0 staticarp

    If anyone has a solution or (better) alternative (appliance) for me, please let me know.
    Thanks in advance,
    Jesper

    ReplyDelete
    Replies
    1. Forgot I was using vSphere 5.1 and seperated all interfaces onto different Virtual Networks on the available vmnic.
      Used PfSense 2.1 Full Install from LiveCD.

      Delete
  15. I now use vyatta, a iOS like software for my routers

    ReplyDelete
    Replies
    1. Hi Mathieu,

      vYatta is great but they removed web-based support from the open source appliance unfortunately.

      Users with less knowledge need to be able to use it also; thus a webinterface is a requirement.

      Thanks,
      Jesper

      Delete
  16. Hi. I am an old folk of m0n0wall and PFSense since firts versions. I used your guide for getting the idea and then made my own deployment. Not very different, just a couple of changes. But OVH thought it was a bad move, so they blocked my machine. Just take care. Seems that new datacenter in Gravelines has some other routing system.

    Task Type
    Incident
    Category
    the whole network
    Status
    CLOSED
    Percent Complete
    100%
    Details
    We have an incident on the dedicated routing in Gravelines.
    Date: Monday, 16 December 2013, 19:08PM
    Reason for closing: Done
    Additional comments about closing: The traffic towards some dedicated server networks was affected by some abnormal traffic on one of the VLAN clients. The machine concerned has been isolated from the network.

    ReplyDelete
  17. Hi!

    Just wanna say thanks, I've been mandated to give SyS hosted solutions (a sub of OVH) a try and come up with a working setup! Following this post - which is thoroughly well done btw! - and you helped me a lot making it work!

    Cheers!

    ReplyDelete
  18. Hi!

    Just wanna say thanks, I've been mandated to give SyS hosted solutions (a sub of OVH) a try and come up with a working setup! Following this post - which is thoroughly well done btw! - and you helped me a lot making it work!

    Cheers!

    ReplyDelete
  19. We are the worlds leading publisher of Squid 'Native ACL' formatted blacklists, that allow for web filtering directly with Squid proxy. Of course we also offer alternative formats for the most widely used third party plugins, such as DansGuardian and Squidguard. And while our blacklists are subscription based, they are as a result of our efforts, of a much higher degree of quality than the free alternatives.

    We hope to serve you,

    --
    Signed,

    Benjamin E. Nichols
    http://www.squidblacklist.org

    ReplyDelete
  20. Bluehost is one of the best web-hosting company for any hosting services you need.

    ReplyDelete
  21. I feel so much smarter now. I like the trend that content is becoming more and more important.I like the trend that content is becoming more and more important.

    VMware Training in Chennai

    ReplyDelete
  22. Gateway Firewall solutions Hyderabad, India | Cloudace
    Cloud Ace Technologies is offering Implementation Services on Cloud Computing, Cloud Services, IT Security, Storage solutionsGateway Firewall solutions Hyderabad,


    ReplyDelete
  23. I read that Post and got it fine and informative.
    vmware courses london

    ReplyDelete
  24. US Open: You need to know the US Open Golf Championship will be the 117th US Open Golf 2017, US Open Schedule to be played June 15–18 at Erin Hills in Erin, Wisconsin, northwest of Milwaukee. For so many years, US Open Live everyone knew what to expect US Open Golf Live and US Open Live Stream. For the second time in three years, the 2017 US Open Golf is headed to a course that has never hosted a major. It will be the first US Golf Open in 25 years that doesn't have the names Tiger Woods or Phil Mickelson among the starting times US Golf Open 2017. Woods is missing all the majors for the second straight year because of a fourth back surgery, which was a month before his DUI arrest in Florida 2017 US Open Golf Live. Mickelson US Open Golf Live Stream, with a record six runner-up finishes in the only major he hasn't won, said he plans to skip because his daughter's high school graduation is the same day as the opening round. US Open TV Coverage and US Open 2017 Results


    US Open Golf 2017
    US Open Golf 2017 Live
    US Open Golf 2017 Live Stream


    US Open Championship
    US Open Championship Live
    US Open Championship 2017

    ReplyDelete
  25. If you want your ex-girlfriend or ex-boyfriend to come crawling back to you on their knees (no matter why you broke up) you must watch this video
    right away...

    (VIDEO) Text Your Ex Back?

    ReplyDelete
  26. Nice post! This is a very nice that I will definitively come back to more times this year! Thanks for informative post. VPS Hosting

    ReplyDelete
  27. Nice post! This is a very nice that I will definitively come back to more times this year! Thanks for informative post. Managed IT Services

    ReplyDelete
  28. Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete
  29. I'm very much inspired when I've visited your blog. Your blog is really informative. Hope you will continue with new article. homeschooling

    ReplyDelete