Private installation with self signed certificate - all traffic from gateway is dropped

TTN Network and Services The Things Stack v3
1

I’ve installed 3.6.2 and the web interface is working, I can create devices and have added a gateway which shows up as on-line.
However, all uplinks from teh gateway are dropped, output from gateway data page message detail as follows:

{
“namespace”: “pkg/gatewayserver”,
“name”: “host_handle”,
“message_format”: “host {host} failed to handle message”,
“attributes”: {
“host”: “cluster”
},
“cause”: {
“namespace”: “pkg/redis”,
“name”: “not_found”,
“message_format”: “entity not found”,
“correlation_id”: “42ffb07aecb6491fa5eb3482537d0289”,
“code”: 5
},
“code”: 5
}

I’ve created the device and set appEUI appID and devEUI, I can see from a terminal on the device it’s sending a join request but the server doesn’t even get that far, just drops teh message.
Any suggestions?

2

The output from a terminal where I’ve run docker-compose up from has the following:

stack_1 | WARN Packet filtered connected_ip=172.18.0.1 connecting_ip=10.10.10.200 error=error:pkg/gatewayserver/io/udp:already_connected (gateway is already connected) gateway_eui=000080029C0FF906 namespace=gatewayserver/io/udp

The Lora gateway is at 10.10.10.200, the 172. address is part of the docker address space

3

The IP address issue is a Docker issue. See https://github.com/TheThingsNetwork/lorawan-stack/issues/2075 for details.

The error in your first post doesn’t look related to the warning in your second post, but it also doesn’t give much information about what could actually be wrong (we don’t actually see what entity it’s looking for).

4

Hi Hylke,
Thanks for the quick reply, rebooting the gateway stops that error message but now I have another,
I have created a device and when it tries to join I see the following in the logs:

stack_1      |  ERROR Failed to deduplicate uplink             dev_eui=3934353770377719 grpc_method=HandleUplink grpc_service=ttn.lorawan.v3.GsNs join_eui=0000000000640000 m_type=JOIN_REQUEST major=LORAWAN_R1 namespace=networkserver phy_payload_len=23 received_at=2020-04-03 21:39:06.538403046 +0000 UTC request_id=01E50ZQCNAAQYT78K1VRHWGGZC
stack_1      |   WARN Finished unary call                      duration=88.14664ms error=error:pkg/redis:store (store error) error_cause=dial tcp 127.0.0.1:6379: connect: connection refused error_correlation_id=d592af72911b444e949e6fdd403bf75b error_name=store error_namespace=pkg/redis grpc_code=Unknown grpc_method=HandleUplink grpc_service=ttn.lorawan.v3.GsNs namespace=grpc request_id=01E50ZQCNAAQYT78K1VRHWGGZC

Any advice would be much appreciated, I feel like I’m very close to having an operational system,
Cheers
H

5

I had originally installed this on Debian 10, thought I’d try Ubuntu 18 and got the same problem with “failed to deduplicate uplink”. I think it must be something to do with the self-signed certificates.
Update, I configured acme and everything seems to be working with a trusted certificate now but still getting the failed to deduplicate error

6

Hello @hmilsom,

By looking at the second log line it seems that Redis is not accessible by the stack:

error=error:pkg/redis:store (store error) error_cause=dial tcp 127.0.0.1:6379: connect: connection refused

Can you confirm that Redis is active and running? (docker-compose ps and/or docker-compose logs redis from directory which contains the docker-compose.yml file)

7

I can’t see anything obvious, the ps command gives:

>     Name                 Command             State              Ports       
    > --------------------------------------------------------------------------------
    > ttnv3_home_cockroach   /cockroach/cockroach   Up             127.0.0.1:26256->26
    > _1                     .sh st ...                            256/tcp, 127.0.0.1:
    >                                                          26257->26257/tcp,  
    >                                                          8080/tcp           
    > ttnv3_home_redis_1     docker-entrypoint.sh   Up             127.0.0.1:6379->637
    >                    redis ...                             9/tcp              
    > ttnv3_home_stack_1     ttn-lw-stack -c        Up (healthy)   0.0.0.0:1700->1700/
    >                    /config/tt ...                        udp, 0.0.0.0:1881->
    >                                                          1881/tcp, 0.0.0.0:1
    >                                                          882->1882/tcp, 0.0.
    >                                                          0.0:1883->1883/tcp,
    >                                                          0.0.0.0:1884->1884/
    >                                                          tcp, 0.0.0.0:1885->
    >                                                          1885/tcp,0.0.0.0:80
    >                                                          ->1885/tcp, 0.0.0.0
    >                                                          :1887->1887/tcp, 0.
    >                                                          0.0.0:8881->8881/tc
    >                                                          p, 0.0.0.0:8882->88
    >                                                          82/tcp, 0.0.0.0:888
    >                                                          3->8883/tcp, 0.0.0.
    >                                                          0:8884->8884/tcp, 0
    >                                                          .0.0.0:8885->8885/t
    >                                                          cp,0.0.0.0:443->888
    >                                                          5/tcp, 0.0.0.0:8887
    >                                                          ->8887/tcp

and redis logs give:

edis_1      | 1:C 06 Apr 2020 04:28:56.120 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1      | 1:C 06 Apr 2020 04:28:56.120 # Redis version=5.9.103, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1      | 1:C 06 Apr 2020 04:28:56.120 # Configuration loaded
redis_1      | 1:M 06 Apr 2020 04:28:56.121 * Running mode=standalone, port=6379.
redis_1      | 1:M 06 Apr 2020 04:28:56.121 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1      | 1:M 06 Apr 2020 04:28:56.121 # Server initialized
redis_1      | 1:M 06 Apr 2020 04:28:56.121 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1      | 1:M 06 Apr 2020 04:28:56.121 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1      | 1:M 06 Apr 2020 04:28:56.122 * DB loaded from append only file: 0.000 seconds
redis_1      | 1:M 06 Apr 2020 04:28:56.122 * Ready to accept connections
redis_1      | 1:signal-handler (1586148868) Received SIGTERM scheduling shutdown...
redis_1      | 1:M 06 Apr 2020 04:54:28.882 # User requested shutdown...
redis_1      | 1:M 06 Apr 2020 04:54:28.882 * Calling fsync() on the AOF file.
redis_1      | 1:M 06 Apr 2020 04:54:28.939 # Redis is now ready to exit, bye bye...
redis_1      | 1:C 06 Apr 2020 04:55:46.505 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1      | 1:C 06 Apr 2020 04:55:46.505 # Redis version=5.9.103, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1      | 1:C 06 Apr 2020 04:55:46.505 # Configuration loaded
redis_1      | 1:M 06 Apr 2020 04:55:46.508 * Running mode=standalone, port=6379.
redis_1      | 1:M 06 Apr 2020 04:55:46.508 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1      | 1:M 06 Apr 2020 04:55:46.508 # Server initialized
redis_1      | 1:M 06 Apr 2020 04:55:46.508 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1      | 1:M 06 Apr 2020 04:55:46.508 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1      | 1:M 06 Apr 2020 04:55:46.655 * DB loaded from append only file: 0.145 seconds
redis_1      | 1:M 06 Apr 2020 04:55:46.655 * Ready to accept connections
8

I upgraded from 3.6.2 to 3.6.3 and all the deduplication errors and join request drops disappeared, everything is working now.

9

This topic was automatically closed after 30 days. New replies are no longer allowed.