It is using the AppKEY in the join request not to encrypt the message, but rather in that the cryptographic checksum (MIC) of a join request is calculated with the AppKEY since the network session key used for that with ordinary traffic doesn’t exist yet.
LoRaWAN is designed so that network and application level secrets can be separate layers of trust. As far as anything the network server knows, this could be a valid join request and gets passed on to the join server/component. But there, it’s found that the message does not validate against the only AppKEY known for that app EUI, because the AppKEY was mistakenly generated anew, rather than whatever AppKEY the device is actually using being found out and entered.