I'm writing this post to highlight a problem I've encountered with User Defined Routes/Route Tables, and their implementation on Azure Stack. I'll not go into detail on when to use UDR's; the official documentation does a great job of that: https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview#custom-routes
Typically you would use a UDR within an Azure Stack tenant's VNet when you want to direct internet bound traffic via a third party network virtualized appliance (e.g. Firewall).
I'll qualify what I mean by 'Internet'. For Public Azure, Microsoft refer to any network within a virtual network as 'internet' that doesn't have a route entry as defined within the SDN (typically route entries will exist for the virtual network/subnets and BGP advertised routes such as peered VNets, ExpressRoute networks, etc). For unknown/external networks, it will forward the request on to the default gateway and via the Azure fabric to pop out onto the internet. If an Azure Stack based VNet encounters an unknown/external network, it will forward the traffic via the SDN to the upstream border switch, which in turn is connected to the corporate network. For disconnected deployments, there may not be a route out to the internet, but there will to the customer network.
Here's a couple of comments I have about this
Microsoft appear to use the same code base for Azure / Azure Stack for UDR's. There is no differentiation between the two platforms, which can cause a problem, as will be highlighted shortly.
For Azure Stack, the term 'Internet' is incorrect in my opinion. Think of 'internet' within Azure Stack as any network external to the appliance.
Now I've described that, here's a scenario and an issue I have encountered.
An Azure tenant deploys a virtual network and they want to control outbound access from the VM's via a firewall appliance. They want to be able to perform remote admin for VM's via a jump server from a secure admin workstation (SAW). They connect to the jump server via a Public IP associated with the jump server NIC. Nothing exotic is being suggested here and is a perfectly normal deployment scenario.
In theory, a UDR is associated to the subnet where the VM's are connected to (10.0.0.128/25) for address prefix 0.0.0.0/0 with next hop as the firewall IP address (10.0.0.4). In order not to cause an issue with routing or spoofing protection within the firewall, another route is added, with the address prefix being 172.16.100.24/32 and the next hop being 'Internet' (remember, Internet should be thought of as external network to the Azure Stack appliance!). Due to longest prefix matching rules being applied, the more specific /32 route will take preference over the /0 route, allowing for the firewall to be bypassed and allowing the SAW to connect via the Public IP.
That's the theory, but here's the problem:
The Corporate network is using an address space that's in the IANA Private use IP range ( https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml ) which is fairly common practice for the majority of enterprise networks. The problem is that I am unable to assign a UDR using an address prefix in any of these private ranges with the next hop as 'internet', as the validation for the route is expecting it to be in Public Address Space.
Exhibit a:
I have tested for the other Private ranges too and get the same outcome.
Obviously for Public Azure, this is not an issue and the validation will do it's job, but that's the problem in using sharing the code with Azure Stack, you don't know what state the customer network is in and what address space they have in use.
I've raised a support case as it will clearly be a problem for some customers, but at this time, I don't have any workaround. This is the case for all Azure Stack versions as of writing (1901).
Now, I'm guessing that by removing the validation step, it would fix the problem. It would be nice to rename the next hop type to 'External Network', or something more appropriate whilst they're at it :)