Network virtualization using Proxmox, Pfsense and VyOS:

By | August 26, 2020

If you are a networking enthusiast or a power user you probably have home network lab like me. Why not vitualize your Pfsense firewall and VyOS router? If you have a dedicated low power pc to be used as a router or firewall, the hardest choices we have to make is the choice of the router or firewall we want to use. Some prefer CLI based distros like VyOS while some prefer Pfsense, OPNSense or other gui based distros. Maybe you want both but have only one box so you have to make a choice. This was the exact same situation I was in a week ago. So I decided to virtualize Pfsense and Vyos for my home using Proxmox PVE 6.2.

The virtualization build Specs:
AMD Athlon 200GE 3.5Ghz 2 core/ 4 thread cpu.
4 GB DDR4 Crucial 2666MHz RAM.
2 x TPLINK TG-3468 v4 Gigabit NICs.
Gigabyte A320 S2H motherboard.
A spare 1TB 7200RPM HDD.
A cheap regular 450 watt PSU from Foxin.

Main hardware components for the build.

Why this build?
As I wanted to spend no more than ₹12,000 i.e., the cost of a standard mikrotik RB3011 I had gone with the above build. Also the main reason for the choice of this CPU is it has some good performance with Pfsense and other networking distros while being very low on power consumption at the minimal pocket pinch. Although I loaded VyOS on a 16GB pendrive and booted the system initially. I soon wanted the intuitive insight on the network that Pfsense offered with it’s GUI, without virtualization it meant that I needed to ditch the VyOS installation. While VyOS is probably the most stable routing operating system with a fast and responsive community it is absolutely a CLI based system. However, on installing Pfsense I realized that Free BSD 11.3 on which Pfsense 2.4.5 is based on has no driver support for this particular tplink NICs. I had gone with an HDD because I didn’t want to spend more. The motherboard has 3 PCI-E slots so I decided to go for it. However, as all the slots share the same PCI-E lane you cannot passthrough NICs on Promox as they are in the same IOMMU group. I would not recommend using the ACS mod to forcefully separate the IOMMU groups as it would not work (tried and tested). It appears that all NICs are on separate IOMMU groups but when you power on a VM it fails to start. These tplink NICs are they cheapest quality NICs with 3 year warranty that you can find for around ₹630

Network Functions Virtualized using proxmox ve

Why virtualize?
In short the answer is not all NOS distros are built the same, some route better while others filter better. Installing just a single os like Pfsense or VyOS on the system may not utilize the resources fully and would restrict me to just one distro. I absolutely love how well Pfsense deals with policy based routing and load balancing while I want to explore and report bugs on VyOS. Also if I wanted to implement any new distro or try it out then that would mean formatting and if anything goes wrong with the new one then I would need to once again format and get back to the old one. With Proxmox I could simply virtualize all of the network functions needed for my home network. Currently I have virtualized a Pfsense VM as my firewall, a VyOS VM as my edge router and two Pi Hole VMs for primary and secondary DNS to implement network wide ad blocking. With Virtualization I could eliminate the driver issue my NICs had with Pfsense bare metal installation.

Proxmox memory and cpu usage.

What about Scalability?
Adding newer network services like an NMS or NAS is relatively easier now. Just add additional drives and some more RAM spin up a VM and we are all good to go. Virtualization of network has helped me eliminate the need to running multiple systems. Now I can take the granular control of firewall on pfsense while enjoying routing features of the VyOS of which I am an absolute fan. We can test any other networking distros by simply spinning up a VM thereby significantly reducing downtime and time spent on getting things up and running. For my present setup with 4GB of RAM I am having Pfsense VM with 2GB memory, VyOS with 1 GB memory and two Debian minimal installations with 768 MB of RAM each. Currently I have about 79 % of memory usage with about 7-9% CPU usage all the time.


Leave a Reply

Your email address will not be published. Required fields are marked *