CMAKE_EXE_LINKER_FLAGS not honored when linking the viewer binary if -DLL_TESTS:BOOL=ON

When configuring with -DLL_TESTS:BOOL=ON linking starts to take several minutes for me now, instead of 8 seconds, because it's using several GB more and causes me to start swapping (at least I don't crash anymore, I fixed that).
The reason for this is that CMAKE_EXE_LINKER_FLAGS is being reset when LL_TESTS is on.