I’m hosting office but is not using rds functionality. Do I have to pay for Rds licenses?

The short answer is YES. By enabling end users to access Office (a graphical user interface), you are providing indirect access to the Windows Server.

SPUR reads that “the delivery of a Desktop Application, such as Office, that is used by providing direct or indirect access to server software that hosts the graphical user interface, such as Windows Server (using the Windows Server Remote Desktop Services functionality or other technology), requires a Windows Server Remote Desktop Services SAL”.