Yes, you did right and understood correctly. (Working example.)
Did you also try to revert the byte order of the values your tried in the “Secret AppKey or AppSKey” field?
I think it’s really about the device using an AppKey that you do not know. (Bugs aside, really all that can cause a different MIC for an OTAA Join Request is a different AppKey; the other data needed to calculate/verify the MIC are taken from the message itself.)
But what is that programming interface you’re showing there? Any other screens in that application?
As for the AES option: I’ve no idea. Maybe the device allows for additional encryption of the application payload. The application payloads of data uplinks and downlinks are already encrypted (using the AppSKey as derived during OTAA). But in LoRaWAN 1.0.x the application server and network server are not 100% separated, so some feel they need more. (I don’t.) Or maybe it’s for non-LoRaWAN operation?
Heads up: the DevNonce, which is also in the Join Request, was hexadecimal 0x008E (decimal 142) for the last message, and 0x0087 in your earlier try. Given only those 2 values, it may be an increasing number, not a random number like LoRaWAN 1.0.x specified. That is not a problem at all, but beware that TTN will remember all DevNonces that the device once used. So, if it ever somehow resets to some factory state and then uses the same sequence of DevNonces, you’ll see OTAA shows "Activation DevNonce not valid: already used". (Workaround in that same topic.)
Also, that sequential DevNonce may suggest it’s a LoRaWAN 1.1.x device. That should be backwards compatible with 1.0.x: the Join Request is the same, and given the Join Accept response the device should understand that the server is on 1.0.x, and fall back to 1.0.x. So, I’m not expecting trouble there, but beware…
Still: OTAA Join first!