How to remove a custom layer?

How is a custom layer supposed to be destructed?
I have a custom layer that implements drawing by overriding IActiveViewEvents::AfterDraw.
The problem is that sometimes when the custom layer is removed from the Scene/Table Of Contents it is still drawn!?

The FinalRelease() method is never called in my custom layer.

1. I would expect it to be ArcScenes responsibility to delete my custom layer when it is removed?; which in turn would call my FinalRelease() method.

2. If my custom layer is not in the Scene/Table Of Contents; how comes it receives IActiveViewEvents::AfterDraw calls?
I would expect ArcScene to draw only layers in the Scene/Table Of Contents.

Thanks in advance for any answers!
Andreas