Only if your goal is to permanently destroy the radio!
Driving I/O voltages above the supply rail is bad, but it’s not as bad as driving the supply rail out of spec, because at least the ATmega I/O’s have some source impedance. The source impedance, and the voltage drop and of the radio’s protection diodes would be why the mis-driven I/O’s are ending up loaded to a voltage intermediate between the radio’s 3.3 volts and the Arduino’s 5v. (In the reverse direction, a 3v3 output really is enough for a 5v ATmega, no issue there)
Given the report of successful peer to peer operation, it’s unlikely that the voltage issue is the immediate cause of difficulty, though it may well cause the radio to fail sooner or later. A quick re-check if the peer-to-peer is still working might be in order. Putting some 2.2k series resistors in the lines would help lessen the stress on the radio, though it’s not really a sufficient fix.
In terms of productive directions for the project, given that an ATmega328 is never going to be better than a squeeze for a LoRaWAN stack, switching to an ARM part that solves both the voltage issue and gives some elbow room is probably a better investment of effort than adding voltage translation. Or one could swap out the voltage regulator on the Nano for a 3v3 one and change the crystal for 8 MHz operation - but that’s a lot of work.
The other useful thing to do for productivity is to get one’s own gateway, as being able to see the raw uplink message attempts is very helpful for distinguishing between situations where the node transmitted with appropriate settings, but data that doesn’t correspond to a valid LoRaWAN uplink from its device settings, vs situations where the node didn’t transmit. And similarly in the downlink situation.
Additionally it should be noted that the 3v3 supply rail output of 5v Arduino’s may have very little current capacity, and may not be able to power a LoRa radio transmission and the amount of current required depends on the radio settings used, so it could even be different between the peer to peer and LMiC attempts. Depending on if you have a “real” or “fake” Arduino that 3v3 supply is probably coming from the on-chip regulator of the USB UART solution, which might be an FTDI or a CH340 or who knows what. That on-chip regulator really isn’t intended to supply anything more than the USB chip and a few supporting components.
Really it would be best to build an all-3v3 system around an ARM MCU, with a “real” 3v3 regulator. Alternately get a real 3v3 regulator for power and some proper voltage conversion for the I/O (dividers should work on the Arduino outputs, don’t really need anything on the inputs if careful to never misconfigure them). My gut suspicion is that your current failure to get packets through is actually something else, but the voltage issues do need to be corrected, too.