You are detaching the interrupt (no real need to do that) but you are not clearing the interrupt flag so when you re-enable the interrupt, an interrupt will occur.
You do not need to attach and detach the interrupt. You can attach it once in setup but disable it. Then when you want to enable it again (just before going to sleep), you clear the interrupt and enable the interrupt.
I can’t remember if the Arduino Mini includes USB, if it does, then you also need to disable the USB peripheral otherwise it will wake up the processor based on normal USB activity.