Just found additional options for the deep-sleep mode. Looking inside the esp_sleep.h the function esp_sleep_pd_config can be used to configure the sleep.
I added this include.
And before every sleep call I set the config.
I reduced the power consumption to 2mA (also powered the esp32 with 3.3V to prevent the blinking led).