I too have noticed that “inactive” layers actually are active from a performance standpoint, or at least that seems to be the case with various scenes. Perhaps it’s a memory issue?
They are not active. There’s an existing thread that covers this false hypothesis in great detail already: Notch Block ~8ms cook time with empty layer selected