Unable to connect Multitech Conduit AEP to TTN V3

I’ve had some progress! However, not solved yet…
(This is on mPower/mLinux 5.3.0 as indirectly mentioned previously)
The main issue seems to be twofold:

For some weird reason, the TTN server (or MultiTech for all I know) decides to change the device EUI into something which does not match the actual EUI. For the purpose of the example, I’ll use the following true device EUI: 00 88 00 00 00 00 AB 99. After communicating with the TTN server, they request info from the EUI 00 00 00 88 00 00 AB 99 instead, i.e moving four zeros from the middle to front. I really don’t understand why. I even tried to manually force set the EUI in the station config and remove the eui-prefix line without any change. But okay, the solution for the time being is to change the EUI in the TTN V3 gateway settings so that it matches the one that results from the server side.

When the above is done, both CUPS and LNS configurations seem to be able to talk to the servers in my case, with their respective server certs and API keys entered correctly. The keys can without any difference be stated without the “Bearer”, but an "Authorization: " in front of the key is strictly necessary! However, the issue is now at the LNS server side where something is refused after first trying to initialise and forward the info from the station. More details follow.

I’ll (try to) attach a printout from the terminal after debugging the basic station manually over ssh. It shows that the CUPS seems to be working as it should, as it is able to collect the LNS stuff (tc.xxx) correctly. The problem seems to start at line 63 where the first error occurs:
2021-02-18 12:45:03.511 [AIO:ERRO] Recv failed: SSL - The peer notified us that the connection is going to be closed
I wonder if it has to do with the EUI issue? Then it tries again and from line 75 onwards new warnings and errors occur. This sequence repeats thereafter.

Anyone who is able to figure out what is causing this problem?

  1. Why is the EUI ending up wrong?
  2. What is the problem with the LNS server connection? CUPS seems to work as intended!

Thanks in advance!
terminal-dump_cupsOK.txt (7.0 KB)

Here’s a screenshot from the TTN V3 live feed
Screenshot 2021-02-18 at 14.29.58

with AEP, i think basic station is not started correctly,

I see the network server settings in the GUI:
image

also in the log I see
image

so I think basic station in AEP 5.3.0 is broken… in mLinux you have more control since you are starting it manually.

@IoT-IES good progress!

@tom-iotexperts I’m not so sure. The basic station can be initiated without any trouble from the AEP web UI here. The problem is that you don’t know what happens, or what goes wrong. So for debugging and figuring out how to correctly initiate it, using the terminal mLinux is superior as you say.

It would, however, be very nice if people from TTN who knows how to initiate this correctly could contribute here. After all they have managed to write a guide which obviously is lacking specific details as none of us here are able to reproduce it.

Here’s a recap so far, for anyone who might not know exactly how to do these things in terminal:
I spent yesterday testing various problems by trial and error. It looks like the certificate one uses is very critical. I ended up using the server’s own certificate, as explained by the multitech page you linked. Also when initiating the basic station from the web UI, all files are deleted/overwritten so that it starts clean everytime. But after I figured out the things mentioned yesterday, I can initiate it to the current partly functioning state (in terms of server connection at all!) from web UI. Now it remains to figure out why it cannot maintain the connection/is rejected by the LNS server…

I used the openssl command to find the certificate which would allow connection:
openssl s_client -showcerts -servername eu1.cloud.thethings.network -connect eu1.cloud.thethings.network:443
This is the CUPS server for Europe (change the server to the one you’re trying to reach), and I copy/pasted the second certificate into the web UI “Server Cert” box. I found that using any of the certificates suggested by TTN did not work, which may be why you’re having trouble connecting as well? And also you should check if the EUI issue is present too!

For debugging, ssh into the mLinux and
sudo -s
for root access. You need this in order to create files at the lora root directory, and it’s convenient not having to sudo with password all the time. Then stop the basic station manually
/etc/init.d/lora-network-server stop
and wait for confirmation. Run it manually to get the errors out (first go to the lora service directory)
cd /run/lora/1
./station
Stop when you have what you need by Ctrl-C. Redo after making changes to the config files. You can list all present files with details by
ls -l
cups.* files are used if the service is initiated in CUPS mode, tc.* files if LNS mode has been chosen (tc files will appear also in CUPS mode, but then they are gathered by the CUPS service). You can change/overwite all the cups/tc files between manual runs to test things. If you reinitiate from web UI, all these will be overwritten again, though.
To start the basic station service again permanently use
/etc/init.d/lora-network-server start
and await confirmation.

Final note on the authorization issues.
You can update the cups/tc.key files in terminal by echoing to the files:
echo "Authorization: NNSXS.GYGYIGHJHJHJHJHJHJHJ......FGHGH" | perl -p -e 's/\r\n|\n|\r/\r\n/g' > cups.key
The perl end-of-file addition here is strictly necessary. Simple \r\n endings as suggested by multitech will fail. The web UI does this in the initialisation, so there you only have to enter the correct CUPS/LNS API KEY in the box with the format
Authorization: NNSXS.GYGYIGHJHJHJHJHJHJHJ......FGHGH

Will update as soon as I figure out anything more. Would appreciate others to do so too!
Screenshot 2021-02-19 at 12.06.26

Basic station configuration has logging options that can be set to output to a file.

See basic station doc.
https://doc.sm.tc/station/conf.html

As @cjhdev pointed out I think the main reason this doesn’t work for the time being is the wrong settings being pushed from the LNS server which overrides the local station.conf. According to MultiTech the clksrc must be 0, not 1 as TTN insists on. I made a new topic specifically for this here: Any way to edit the TTNv3 gateway global_conf.json?

In the meantime, stick to mp_pkt_fwd on TTNv2, or use the default (legacy) PACKET FORWARDER mode in the AEP webUI for either TTNv3 or “legacy” gateways (EUI-mode) on TTNv2. The latter works as long as you make sure the EUI in the TTNvX matches the LORA module EUI. Router address is router.eu.thethings.network for TTNv2 and eu1.cloud.thethings.network for TTNv3 (change eu for other regions…). Ports are 1700 up/down in both cases. Gateway keys are irrelevant for the legacy forwarder.
Untitled picture

I am able to connect the TTN v3 using CUPS in mPower 5.3.0 on Conduit following these instructions.
https://www.thethingsindustries.com/docs/gateways/lora-basics-station/cups/

The gateway key needs to be configured using the ttn-lw-cli.

The basic station on Conduit will ignore the clksrc setting sent from TTN. We build from this forked repo.

1 Like

Hi Everyone

As to @jreiss recommendation, I followed the provided CUPS guide and managed to connect the AEP Conduit to TTN V3 :clap:

One thing to notice, you need to use the default station config by multitech and not the station config settings you can download from TTN. This same config works for any channel plan when using CUPS as it will get the channel plan from the LNS.

if you overwrite it or don’t have it, you can grab it at the top of this article:
http://www.multitech.net/developer/software/lora/running-basic-station-on-conduit/

direct link for MTCDT station config

Few other tips:

Good luck and hopefully you will now have a less frustration migration of Multitech Conduit to Basic Station than I did

4 Likes

Hi Tom,

Thank you for sharing your findings. I have tried to follow your tips, but connection to the CUPS server fails. It seems that https://eu1.cloud.thethings.network:443/update-info/ doesn’t exist, as I get a 404 (verified both by “curl -v https://eu1.cloud.thethings.network:443/update-info/” and by enabling logging in station config)

2021-04-21 13:40:09.728 [CUP:INFO] Starting a CUPS session now.
2021-04-21 13:40:09.728 [CUP:INFO] Connecting to CUPS ... https://eu1.cloud.thethings.network:443 (try #34)
2021-04-21 13:40:09.730 [any:INFO] ./cups.trust:
cert. version     : 3
serial number     : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00
issuer name       : C=US, O=Internet Security Research Group, CN=ISRG Root X1
subject name      : C=US, O=Internet Security Research Group, CN=ISRG Root X1
issued  on        : 2015-06-04 11:04:38
expires on        : 2035-06-04 11:04:38
signed using      : RSA with SHA-256
RSA key size      : 4096 bits
basic constraints : CA=true
key usage         : Key Cert Sign, CRL S2021-04-21 13:40:09.730 [AIO:INFO] cups has no cert configured - running server auth and client auth with token
2021-04-21 13:40:09.844 [CUP:VERB] Retrieving update-info from CUPS https://eu1.cloud.thethings.network:443...
2021-04-21 13:40:10.318 [any:VERB] Failed to retrieve TCURI from CUPS: (404) Not Found
2021-04-21 13:40:10.318 [AIO:DEBU] [4] HTTP connection shutdown...
2021-04-21 13:40:10.319 [CUP:INFO] Interaction with CUPS failed - retrying in 1m
2021-04-21 13:40:10.319 [TCE:INFO] Starting TC engine
2021-04-21 13:40:10.320 [TCE:ERRO] No TC URI configured
2021-04-21 13:40:10.320 [TCE:INFO] Router rejected or retry limit reached. Invoking CUPS.
2021-04-21 13:40:10.320 [TCE:INFO] Terminating TC engine
2021-04-21 13:40:10.321 [CUP:INFO] Starting a CUPS session in 60 seconds.

I see other posts in the forum mentioning that CUPS isn’t implemented yet for TTNv3

Hi @chrille

Does your basic station config looks like this?

(replace URI: with https://eu1.cloud.thethings.network:443)

image

Hi @tom-iotexperts
Yes, this is a screenshot of my config. (gateway key truncated in screenshot - I tried both with, and without, “Bearer”)
Skærmbillede 2021-04-21 kl. 16.39.16

The gateway is Multitech conduit MTCDT-210A with firmware 5.3.3 (latest available for download)
I noticed that the certificate in your server config doesn’t matche the one from https://letsencrypt.org/certs/isrgrootx1.pem - however, as there’s no TLS issues I don’t think the cert is the issue

Following up to myself…
After changing EUI in the TTN console from “00 80 00 00 00 00 C2 F5” to “00 00 00 80 00 00 C2 F5” CUPS actually works and continues to LNS.However, now the initialization of the radio fails.

2021-04-21 19:05:51.283 [TCE:INFO] Connecting to INFOS: wss://eu1.cloud.thethings.network:8887
2021-04-21 19:05:51.791 [TCE:INFO] Infos: 80::c2f5 muxs-::0 wss://eu1.cloud.thethings.network:8887/traffic/eui-000000800000C2F5
2021-04-21 19:05:51.791 [AIO:DEBU] [4] ws_close reason=1000
2021-04-21 19:05:51.791 [AIO:ERRO] Recv failed: SSL - The peer notified us that the connection is going to be closed
2021-04-21 19:05:51.791 [AIO:DEBU] [4] WS connection shutdown...
2021-04-21 19:05:51.793 [any:INFO] ./tc.trust:
cert. version     : 3
serial number     : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00
issuer name       : C=US, O=Internet Security Research Group, CN=ISRG Root X1
subject name      : C=US, O=Internet Security Research Group, CN=ISRG Root X1
issued  on        : 2015-06-04 11:04:38
expires on        : 2035-06-04 11:04:38
signed using      : RSA with SHA-256
RSA key size      : 4096 bits
basic constraints : CA=true
key usage         : Key Cert Sign, CRL Sig2021-04-21 19:05:51.794 [AIO:INFO] tc has no cert configured - running server auth and client auth with token
2021-04-21 19:05:51.852 [TCE:VERB] Connecting to MUXS...
2021-04-21 19:05:52.334 [TCE:VERB] Connected to MUXS.
2021-04-21 19:05:52.390 [S2E:WARN] Feature not supported in production level code (router_config) - ignored: nocca
2021-04-21 19:05:52.390 [S2E:WARN] Feature not supported in production level code (router_config) - ignored: nodc
2021-04-21 19:05:52.390 [S2E:WARN] Feature not supported in production level code (router_config) - ignored: nodwell
2021-04-21 19:05:52.391 [RAL:WARN] Ignoring unsupported/unknown field: antenna_gain
2021-04-21 19:05:52.391 [RAL:INFO] Lora gateway library version: Version: 5.0.1-mts-7;
2021-04-21 19:05:52.444 [RAL:VERB] Connecting to device: /dev/spidev0.0
2021-04-21 19:05:52.444 [RAL:DEBU] SX130x txlut table (16 entries)
2021-04-21 19:05:52.444 [RAL:VERB] SX1301 txlut  0:  dig_gain=0 pa_gain=0 dac_gain=3 mix_gain=11 rf_power=-6
2021-04-21 19:05:52.444 [RAL:VERB] SX1301 txlut  1:  dig_gain=0 pa_gain=0 dac_gain=3 mix_gain=13 rf_power=-3
2021-04-21 19:05:52.445 [RAL:VERB] SX1301 txlut  2:  dig_gain=0 pa_gain=1 dac_gain=3 mix_gain=9 rf_power=0
2021-04-21 19:05:52.445 [RAL:VERB] SX1301 txlut  3:  dig_gain=0 pa_gain=1 dac_gain=3 mix_gain=10 rf_power=3
2021-04-21 19:05:52.445 [RAL:VERB] SX1301 txlut  4:  dig_gain=0 pa_gain=1 dac_gain=3 mix_gain=12 rf_power=6
2021-04-21 19:05:52.445 [RAL:VERB] SX1301 txlut  5:  dig_gain=0 pa_gain=2 dac_gain=3 mix_gain=10 rf_power=10
2021-04-21 19:05:52.445 [RAL:VERB] SX1301 txlut  6:  dig_gain=0 pa_gain=2 dac_gain=3 mix_gain=11 rf_power=11
2021-04-21 19:05:52.445 [RAL:VERB] SX1301 txlut  7:  dig_gain=0 pa_gain=2 dac_gain=3 mix_gain=11 rf_power=12
2021-04-21 19:05:52.445 [RAL:VERB] SX1301 txlut  8:  dig_gain=2 pa_gain=2 dac_gain=3 mix_gain=12 rf_power=13
2021-04-21 19:05:52.445 [RAL:VERB] SX1301 txlut  9:  dig_gain=0 pa_gain=2 dac_gain=3 mix_gain=13 rf_power=14
2021-04-21 19:05:52.445 [RAL:VERB] SX1301 txlut 10:  dig_gain=0 pa_gain=2 dac_gain=3 mix_gain=15 rf_power=16
2021-04-21 19:05:52.445 [RAL:VERB] SX1301 txlut 11:  dig_gain=0 pa_gain=3 dac_gain=3 mix_gain=10 rf_power=20
2021-04-21 19:05:52.445 [RAL:VERB] SX1301 txlut 12:  dig_gain=0 pa_gain=3 dac_gain=3 mix_gain=12 rf_power=23
2021-04-21 19:05:52.445 [RAL:VERB] SX1301 txlut 13:  dig_gain=0 pa_gain=3 dac_gain=3 mix_gain=13 rf_power=25
2021-04-21 19:05:52.446 [RAL:VERB] SX1301 txlut 14:  dig_gain=0 pa_gain=3 dac_gain=3 mix_gain=15 rf_power=26
2021-04-21 19:05:52.446 [RAL:VERB] SX1301 txlut 15:  dig_gain=0 pa_gain=3 dac_gain=3 mix_gain=15 rf_power=27
2021-04-21 19:05:52.446 [RAL:VERB] SX1301 rxrfchain 0: enable=1 freq=867.5MHz rssi_offset=-166.000000 type=2 tx_enable=1 tx_notch_freq=0
2021-04-21 19:05:52.446 [RAL:VERB] SX1301 rxrfchain 1: enable=1 freq=868.5MHz rssi_offset=-166.000000 type=2 tx_enable=0 tx_notch_freq=0
2021-04-21 19:05:52.446 [RAL:VERB] SX1301 ifchain  0: enable=1 rf_chain=1 freq=-400000 bandwidth=0 datarate=0 sync_word=0/0
2021-04-21 19:05:52.446 [RAL:VERB] SX1301 ifchain  1: enable=1 rf_chain=1 freq=-200000 bandwidth=0 datarate=0 sync_word=0/0
2021-04-21 19:05:52.446 [RAL:VERB] SX1301 ifchain  2: enable=1 rf_chain=1 freq=0 bandwidth=0 datarate=0 sync_word=0/0
2021-04-21 19:05:52.446 [RAL:VERB] SX1301 ifchain  3: enable=1 rf_chain=0 freq=-400000 bandwidth=0 datarate=0 sync_word=0/0
2021-04-21 19:05:52.446 [RAL:VERB] SX1301 ifchain  4: enable=1 rf_chain=0 freq=-200000 bandwidth=0 datarate=0 sync_word=0/0
2021-04-21 19:05:52.447 [RAL:VERB] SX1301 ifchain  5: enable=1 rf_chain=0 freq=0 bandwidth=0 datarate=0 sync_word=0/0
2021-04-21 19:05:52.447 [RAL:VERB] SX1301 ifchain  6: enable=1 rf_chain=0 freq=200000 bandwidth=0 datarate=0 sync_word=0/0
2021-04-21 19:05:52.447 [RAL:VERB] SX1301 ifchain  7: enable=1 rf_chain=0 freq=400000 bandwidth=0 datarate=0 sync_word=0/0
2021-04-21 19:05:52.447 [RAL:VERB] SX1301 ifchain  8: enable=1 rf_chain=1 freq=-200000 bandwidth=2 datarate=2 sync_word=0/0
2021-04-21 19:05:52.447 [RAL:VERB] SX1301 ifchain  9: enable=1 rf_chain=1 freq=300000 bandwidth=3 datarate=50000 sync_word=0/0
2021-04-21 19:05:52.447 [RAL:VERB] SX130x LBT not enabled
2021-04-21 19:05:52.447 [RAL:VERB] Station device: /dev/spidev0.0 (PPS capture enabled)
2021-04-21 19:05:52.450 [RAL:ERRO] Concentrator start failed: lgw_start
2021-04-21 19:05:52.450 [RAL:ERRO] ral_config failed with status 0x08
2021-04-21 19:05:52.450 [any:ERRO] Closing connection to muxs - error in s2e_onMsg
2021-04-21 19:05:52.450 [AIO:DEBU] [4] ws_close reason=1000
2021-04-21 19:05:52.451 [AIO:DEBU] Echoing close - reason=1000
2021-04-21 19:05:52.504 [AIO:DEBU] [4|WS] Server sent close: reason=1000
2021-04-21 19:05:52.504 [AIO:DEBU] [4] WS connection shutdown...
2021-04-21 19:05:52.505 [TCE:VERB] Connection to MUXS closed in state -1
2021-04-21 19:05:52.505 [TCE:INFO] INFOS reconnect backoff 10s (retry 1)

So now I’m stuck at the same place as IoT-IES

Skærmbillede 2021-04-21 kl. 21.19.59

[edit - next day]

I found this post MultiTech Developer Resources » Topic: LoRa Basics Station with USB (MTAC-LORA-1.0) Lora card - it turns out that the Lora module in my conduit is the old model (MTAC-LORA-1.0) that simply isn’t supported for basic station mode. So I either need to spend 200 dollars on the new model, or look into the UDP packet forwarder.
Anyway, the original issue with CUPS and TTNv3 seems to be solved.

1 Like

Good catch, so UDP works just fine really, i wouldn’t invest to upgrade the radio just because of basic station.

1 Like

Hello.
I am having MTCDT-LEU1-210A-EU-GB problem.I set up a simple station 2.0.5-1-r2.0 says RUNNING but Disconnected in TTN V3.Never seems connected in TTN V3.How can I fix?

@bekir, you need to configure CUPS and LNS keys on TTS. Have a read here: LoRa Basics™ Station | The Things Stack for LoRaWAN

@ElectronicallyE
hey you did a very nice summary of the setup in your post, you forgot to mention in there the LNS KEY creation which is part of the process.

also seems that TTN updated the documentation on this step and now you can add this LNS API key in the GW General Settings page

where before this was done only with the CLI tool
image

question: can this LNS key be created once for all the GWs in the Personal API keys or in Org keyst, or does it have to be created per gateway?

another comment: if you are using The Things Industries or the things stack, do use this command to grab the cert:
openssl s_client -showcerts -servername au1.cloud.thethings.network -connect au1.cloud.thethings.network:443

1 Like

i was about to give up as I faced the same issue as bakir. i did all the steps mentioned here but the

gateway still disconnected(and it is still ).

image

however, just before I was about to delete the gateway and go back to TTN V2, I suspected an old

issue that used to happens in V2, which was a disconnected state on the gateway console but the

application console still receives live data. I configured an application and a device (with a

disconnected gateway status on the console). surprisingly, the same issue seems to be still

happening. i was receiving data from my sensor. unfortunately, TTN removed the gateway ID from the

meta-data hence I suspected that maybe another gateway around caught my packet so I disconnected

my gateway and retransmit again from the sensor but nothing received on the application’s live data.

So I guess that V3 maybe still suffers from that old problem.

image

It does that for data caught by a V2 GW coming through PacketBroker into V2 applications…so one would suspect your GW was/still is registered in V2 and not yet on V3? (If so DON’T delete from V2 just in case - just change key to stop it being picked up in V2 via console)

Thank you Jeff, The Gateway was deleted but the sensor was registered on V2…but I think this is not related to the fact that the GW is in a disconnected state, No?

To use a basic station, it must be 1.5 lora hardware and above.Unfortunately my 1.0