I confirm that I sent an application command in order to modify the reporting time of the node.
I agree, the console could really help in debug, but sometimes (… when I need it 
) it diseappears.
In the meantime I will continue to collect the downlink msg with mosquitto and let’s see if something will arrive from TTN backend …
Gianluigi
Given my previous answer (which I posted while you writing yours) I’d say it’s banned until deleted and re-created. The maximum failures is 3. But again: I think it’s only banned when in the next uplink it does acknowledge it received the ADR request, but then tells TTN it refused one of the settings, being data rate, power and/or channels. In other words: if it could not comply to TTN’s request, in which case repeating it makes no sense. This would better be described as “rejected” rather than “failed”.
Keep the gateway’s Traffic page in TTN Console open to be able to see the Trace part when clicking an uplink or downlink!
Beware:
And:
In your case, the latter seems okay, as the downlinks are in RX2, 869.525/SF9, which is different from the uplink channel anyway. The first might apply though, making the ADR downlink not being handled on the device.
Are all nodes using OTAA? Or when using ABP, did you adjust the RX2 settings to match the non-standard TTN settings? (I guess this is fine, as it would apply to both devices, and one works just fine.)
Oh, and all of the above might not apply to you, as we already determined that at least one of the downlinks did not hold any ADR command at all…
The gw is about 15 meter from the nodes and I’m only using OTAA.
I’m attaching the trafic log files about the 10609d node both in up and down direction, with some comments inside.
No downlink messagge from yesterday midday, till this morning when I moved the nodes from home to work site. When I arrived at work, one downlink was sent by TTN, but SF is always 12.
1060d9-up-18022019.txt (127.8 KB)
1060d9-down-18022019.txt (1.8 KB)
I try something more with another node, 1060bd.
Yesterday this node opeartes with a SF7, then I shielded it for all the night and this morning I remove the shield and I found it with a SF12.
Then TTN start to send downlink msg every time the node send an uplink (every hour).
This is the mosquitto downlink log from yesterday till now:
Feb 17 09:18:27 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaAFwACYNKhHPXwO+Zs3isTHM1fx1Ffv0A=”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:2,“payload_raw”:“AQcBAQcIBuMCBwgDXwkADw==”},“gateway_id”:“eui-1dee0eb900028599”,“config”:{“modulation”:“LORA”,“data_rate”:“SF7BW125”,“airtime”:66816000,“counter”:23,“frequency”:867700000,“power”:14}}
Feb 18 06:49:52 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaFGAADVf8AAzz/Bqw=”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee03a372f1126a”,“config”:{“modulation”:“LORA”,“data_rate”:“SF9BW125”,“airtime”:164864000,“counter”:24,“frequency”:869525000,“power”:27}}
Feb 18 07:49:39 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaKGQAFA9KthANV/wADLIV4/Q==”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee03a372f1126a”,“config”:{“modulation”:“LORA”,“data_rate”:“SF12BW125”,“airtime”:1482752000,“counter”:25,“frequency”:869525000,“power”:27}}
Feb 18 08:49:42 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaKGgAFA9KthANV/wADYCLDlQ==”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee1a0a6e94d4bf”,“config”:{“modulation”:“LORA”,“data_rate”:“SF12BW125”,“airtime”:1482752000,“counter”:26,“frequency”:869525000,“power”:27}}
Feb 18 09:49:50 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaKGwAFA9KthANV/wADJbGK9Q==”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee1a0a6e94d4bf”,“config”:{“modulation”:“LORA”,“data_rate”:“SF12BW125”,“airtime”:1482752000,“counter”:27,“frequency”:869525000,“power”:27}}
Feb 18 10:49:41 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaKHAAFA9KthANV/wADHOnA3Q==”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee1a0a6e94d4bf”,“config”:{“modulation”:“LORA”,“data_rate”:“SF12BW125”,“airtime”:1482752000,“counter”:28,“frequency”:869525000,“power”:27}}
Feb 18 11:50:04 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaKHQAFA9KthANV/wADtk6xvQ==”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee1a0a6e94d4bf”,“config”:{“modulation”:“LORA”,“data_rate”:“SF12BW125”,“airtime”:1482752000,“counter”:29,“frequency”:869525000,“power”:27}}
Feb 18 12:49:36 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaKHgAFA9KthANV/wADcpP3+A==”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee1a0a6e94d4bf”,“config”:{“modulation”:“LORA”,“data_rate”:“SF12BW125”,“airtime”:1482752000,“counter”:30,“frequency”:869525000,“power”:27}}
Feb 18 13:49:56 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaNHwACFwEFA9KthANV/wADQ9uFew==”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee1a0a6e94d4bf”,“config”:{“modulation”:“LORA”,“data_rate”:“SF12BW125”,“airtime”:1482752000,“counter”:31,“frequency”:869525000,“power”:27}}
I’ve also catched the last one downlink (13:49:56) with the TTN console, that shows the ADR commands:
I don’t know if the link-adr commands are correct, from the mosquitto log there is a “config” payload that I don’t know if refers to the current condition or represent a configuration command.
In case the adr command are really trying to lower the SF, it could be that the problem is the gateway or the node.
Gianluigi
Given the frequency 869.525, the downlinks are all in RX2, except for the very first one, which is in RX1 (and is the Join Accept). Of those RX2 downlinks, only the first is sent at the TTN-specific SF9 (as expected), all others use the LoRaWAN default SF12 (as a workaround).
Now, an example has proved that, when using RX2, TTN surely is smart enough to repeat ADR on SF12 if nothing changed when the node ignored the ADR downlink in the documented SF9. The same can been seen in the code. But I feel TTN should not keep using SF12 for ADR in RX2 then†; assuming your node is using the TTN specific settings that it got in the EU868 OTAA Join Accept, it will never receive the RX2 downlinks when TTN uses SF12…
On the other hand, maybe the bug is that it should not repeat forever, but only once. And maybe it uses SF9 again when the node sets ADRAckReq at some later time again. And the other nodes do accept the RX2 SF9 downlink?
That’s the gateway’s configuration for the downlink. Like frequency, data rate and power that is to be used for the transmission. It’s not some configuration payload that is sent to the node.
The data sent to the node is taken from, e.g., "payload": "YOEvASaFGAADVf8AAzz/Bqw=" for the SF9 downlink, which indeed is a MAC command in FOpts = 0355FF0003, which according to the 1.0.2 specification and the EU868 regional parameters decodes to:
And the first SF12 downlink, which your node will not have received, gives FOpts = 0503D2AD84 0355FF0003, which indeed first tells the node about the TTN specific RX2 settings:
…followed by the LinkADRReq MAC command 0x0355FF0003 as described above.
So, all above aside: why didn’t that specific node receive or process the SF9 downlink? (As other nodes do process it, so certainly receive it, I doubt the gateway is to blame.)
† It seems that for OTAA, this was fixed in October 2019: RX2 fallback leads to endless downlinks · Issue #767 · TheThingsArchive/ttn · GitHub, for devices that joined after that fix.
Ah, the code seems to show that this number is increased when packet loss has been detected. Surely shielding all night has made TTN detect that. In time, this number will decrease again…
Thanks Arjan fo your detailed and really interesting explanation, I’ll let you know about any news.
Gianluigi
In order to identify where is the problem, I’m trying debugging the connection with a ThingsNode.
When I download a byte on the port 0 of the node, I’m able to see it in the log of the node, even if the msg will be reported related to port 2.
When I send to port 2 it’s correctly reported oin port 2
Setting the adr on in the node (ttn.reset(true)), when the msg cnt is 64, TTN correctly send an adr command to the node (14:01:42), but there isn’t any trace of this msg in the node debug console.
Considering that in the node code I’m using the method “ttn.onMessage(message)” for receiving the download msg, is it correct that it catchs only the application level or should it also get the mac msg ?
Gianluigi
That’s not supported; port 0 is reserved for downlinks without any application payload. That is: downlinks that only hold network MAC commands.
I assume TTN actually fixes this for you. It surely does that if it includes a MAC command.
Yes. Any MAC commands will be handled by the RX2xx3 chip.
Thanks Arjan,
so if I correctly understand, I’m not able to verify if the gateway forward the adr request to the node by mac command.
Anyway at the beginning of the log the mac command sent by the node are reported.
The gateway won’t change whatever is in the downlink. It cannot even read all its contents and surely cannot change the MIC, as it does not know the secrets.
But: you can decipher the downlink using the gateway’s log, or the gateway’s Traffic page in TTN Console. So if you see that a downlink was received in the node’s log, you’ll also know if that included any MAC commands. Of course, you won’t know if the RN2xx3 handled those…
You started with:
Maybe they actually have different versions of the RN2xx3 chips, or their firmware?
The version 1.0.4 you show in your last screenshot is slightly outdated but probably good enough.
Sorry, may be I wasn’t clear enough.
The last test wasn’t carried out with one of the four nodes that give me the original problem, because I could’t conect with them to see theirs log.
For check if the adr commnad sent by TTN reach the node, I used another lora node (“The Things Node”).
In the Arduino IDE debug console of this node, I’m able to see the downlink msg that I manually sent from the TTN console, but when the TTN console shows an adr command sent from TTN to “The Things Node” after 64 uplinks, there is not trace of this msg in the Arduino IDE debug console of the node.
Yup, it depends on the LoRaWAN implementation if you can see any MAC commands in the log. For example, for LMIC on Arduino one can enable verbose logging to see ADR MAC commands.
It’s probable that the mac msg are not logged on the node console.
I set SF 8 and enabled adr and after 64 msg it correctly swith from SF 8 to 7 even if the mac command it’s not reported in the log.
Always using the same setup based on “The Things Node” I investigate the behaviour with different SF initial setup obtained defining the “sf” with the method:
TheThingsNetwork ttn(loraSerial, debugSerial, freqPlan, sf);
Starting from SF7 to 10 included, both the Join Request and Accept are always with the SF set up by the above instruction.
These are the output of the TTN gateway console and the Things Node debug console:
…
but if I set SF to 11 or 12, the Join Accept answer is always with SF12
For these two last conditions with SF11 and 12, the node try 3 times to join without any success and then try to send messagge but of course it was not able to.
I don’t know the detail of the lorawan at this level, it’s normal this behaviour in your opinion ?
Thanks,
Gianluigi
P.S.
One more think that seems confirmed is that both Things Node and another type of adr functioning node set the first uplink with counter 0, while the four nodes that give adr problems start with cnt 1.
Please search the forum for rn2483 and SF12 and take a look at the discussions regarding the use of RX2.
With regards to the failing joins at SF11 and SF12, check this forum message and the ones below.
Many thanks Jac,
so I think that testing with “The THings Node” doesn’t help me so much to identify the problem I have with ADR in the chain that includes TTN, Lorank8 gateway and the 4 SensingLab node of the same type.
So, I’m now trying another way.
I changed the packet forwarder in the lorank8 gateway to “Loriot” in order to test how these SensingLab nodes operate with this configuration.
I switched off and on two of this nodes and after 13 uplink, they finally changed from SF12 to SF11 …
The online Loriot log shows these events:
I’m also attaching the lorank8 gateway daemon,log section related to the above events.
I didn’t find the events related to the second node (08:33:42 above) in this log file, but the first one (08:33:14) is reported.
gw-log-loriot-pkt-fwd.txt (6.2 KB)
So it seems that the solution “Loriot+Loriot pkt-fwd” gives some ADR positive results for these nodes type.
I see in the forum some discussion about the poly pkt-fwd, it could be useful to test the original Semtech one ?
Gianluigi
Very much agreed, very confusing… I think the same would apply to trying different packet forwarders on the same network. I feel we already established that TTN does send the ADR commands, in RX2, on SF9, which works for some node(s), but:
Can you repeat this behavior (so: TTN sending an ADR downlink in RX2/SF9, that specific node consistently not changing its settings, TTN re-trying in RX2/SF12 which the node should not receive to start with) if you decode some more uplinks/downlinks? Or did that happen only once?
To ensure TTN does not use old ADR statistics, this might require you to re-register all nodes in TTN Console as maybe TTN preserves the statistics for a new OTAA Join. I’ve no idea; we could peek into the code. Or maybe just temporarily changing the AppKey will clear the statistics, like it clears the DevNonce then. Registration and temporarily changing the AppKey could be automated using the command line ttnctl.
What does this mean? Does that make them lose all their state? (Given the low frame counters, I assume they do.) Or does this only imply that they’ve not been sending for some time? For ADR, TTN doesn’t care how often a node sends; it cannot know if a node is just sleeping, or has been switched off. Unless, of course, the node’s internal state is lost. Like: does it re-join after a power cycle? Does it start at a specific SF after restarting?
But so does TTN, right, for some but not all nodes?
If one node is having problems then it seems to me the nodes are just not 100% the same. Different software, different radio module (RN2xx3?), different firmware in that radio module. Just return the faulty one?
Thanks Arjan,
in order to contains the confusion, I try to summarize the current situation:
First of all, some preliminary clarifications:
For simplicity let’s consider two test scenarios, both related only to the SensingLab nodes:
I hope this could help,
Gianluigi