Just an idea. In my SwayWM setup, a) I run every application in its own cgroup (via systemd-run), and b) I have a daemon that subscribes to window and workspace events, gets the current tree, figures out the PIDs corresponding to invisible vs visible windows, figures out the cgroups corresponding to those PIDs, and then uses cgroup freezer file to freeze / unfreeze them. Other than a few hard-coded exceptions that I want to run while invisible (IRC client etc), any invisible application is transparently frozen and unfrozen.
For "single-instance applications" like firefox, launching a new firefox while the existing one is frozen will hang, so instead of launching /usr/bin/firefox directory I have an intermediate ~/.local/bin/firefox script that unfreezes the firefox cgroup and then exec's /usr/bin/firefox.
Of course if at least one FF window is visible then this doesn't help with your problem, since the cgroup as a whole will be unfrozen. It only helps if none of the FF windows are visible.
For "single-instance applications" like firefox, launching a new firefox while the existing one is frozen will hang, so instead of launching /usr/bin/firefox directory I have an intermediate ~/.local/bin/firefox script that unfreezes the firefox cgroup and then exec's /usr/bin/firefox.
Of course if at least one FF window is visible then this doesn't help with your problem, since the cgroup as a whole will be unfrozen. It only helps if none of the FF windows are visible.