Is a VPN or Express Route mandatory to establish on-premises application connectivity?

VPN or Express Routes are required only in scenarios involving Hybrid connectivity back to on premises environments where servers or applications connect to Azure on ports which cannot be exposed to the internet. Imagine DC to DC replication traffic, a file share access over SMB protocol (Server Message Block).

If the connection across on-prem servers and Azure components are over secure ports, then this can potentially happen over the Internet. Alternate options like Biztalk services can serve to decouple applications components and help to span the stack across on-prem and Azure over Internet.