Unable to get data from devices in TTS

[Kevin192291]

Hello all.
I have been using the things stack that I deployed on AWS for a few years now without any problems.
I now would like to remove the cost of AWS, and host myself. I get TTS setup much in the same way that I did on aws using docker without much issue, though while hosting myself, I need to use a reverse proxy. To the best of my knowledge this process went smooth and without issue. The entire setup probably took somewhere between 15-20 minutes.

I removed 2 of my gateways from AWS and connected them to my local instance and they do seem to be connected correctly:

Here is where things get a bit strange:

This is an image of what I see after sitting on the gateway page for abuot 5 minutes. the “Last seen” counter keeps resetting, however I do not see any data. very strange.

Then If I connect my computer via VPN to the network that the server hosting TTS is on, i see this:

we have data!

but… why are we not seeing data without the VPN? Could be a port issue?
Here is the data from the gateway message:

IP Address 10.0.1.69 is the address of the gateway running on the same network as the server.

Here is data from the 2nd gateway:

That gateway’s IP address is elsewhere on the internet and I see the same stuff, I can view data when on the VPN, but see nothing other than the couner reset when not. This makes me think again that it is a port issue.

the interesting thing is that the gateway does show a device that has sent data, however that device data never makes it to the application that it is registered within.

Here is my nginx reverseProxy config, it is very simple:

server {
listen 443 ssl;
server_name <website name here!!!>
location / {
proxy_pass http://10.0.1.10;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}

and my docker compose file:

version: ‘3.7’
services:

postgres:
# In production, replace ‘latest’ with tag from Docker
image: postgres:latest
restart: unless-stopped
environment:
- POSTGRES_PASSWORD=root
- POSTGRES_USER=root
- POSTGRES_DB=ttn_lorawan
volumes:
- ${DEV_DATA_DIR:-.env/data}/postgres:/var/lib/postgresql/data
ports:
- “127.0.0.1:5432:5432”

redis:
# In production, replace ‘latest’ with tag from Docker
image: redis:latest
command: redis-server --appendonly yes
restart: unless-stopped
volumes:
- ${DEV_DATA_DIR:-.env/data}/redis:/data
ports:
- “127.0.0.1:6379:6379”

stack:
# In production, replace ‘latest’ with tag from Docker
image: thethingsnetwork/lorawan-stack:latest
entrypoint: ttn-lw-stack -c /config/ttn-lw-stack-docker.yml
command: start
restart: unless-stopped
depends_on:
- redis
- postgres
volumes:
- ./blob:/srv/ttn-lorawan/public/blob
- ./config/stack:/config:ro
- ./public:/public:ro
# If using Let’s Encrypt:
#- ./acme:/var/lib/acme
environment:
TTN_LW_BLOB_LOCAL_DIRECTORY: /srv/ttn-lorawan/public/blob
TTN_LW_REDIS_ADDRESS: redis:6379
TTN_LW_IS_DATABASE_URI: postgres://root:root@postgres:5432/ttn_lorawan?sslmode=disable

ports:
  # If deploying on a public server:
  - "80:1885"
  - "443:8885"
  - "1881:1881"
  - "8881:8881"
  - "1882:1882"
  - "8882:8882"
  - "1883:1883"
  - "8883:8883"
  - "1884:1884"
  - "8884:8884"
  - "1885:1885"
  - "8885:8885"
  - "1886:1886"
  - "8886:8886"
  - "1887:1887"
  - "8887:8887"
  - "1700:1700/udp"

# If using custom certificates:
secrets:
  - ca.pem
  - cert.pem
  - key.pem

If using custom certificates:

secrets:
ca.pem:
file: ./ca.pem
cert.pem:
file: ./cert.pem
key.pem:
file: ./key.pem

and here is my ttn-lw-stack-docker file:

Identity Server configuration

Email configuration for “lorawan-01.MY DOMAIN NAME”

is:
email:
sender-name: “SECRET”
sender-address: “noreply@lorawan-01.MY DOMAIN NAME”
network:
name: “network”
console-url: “https://lorawan-01.MY DOMAIN NAME/console”
identity-server-url: “https://lorawan-01.MY DOMAIN NAME/oauth”

# If sending email with Sendgrid
# provider: sendgrid
# sendgrid:
#   api-key: '...'              # enter Sendgrid API key

# If sending email with SMTP
# provider: smtp
smtp:
  address:  'smtp.office365.com'             # enter SMTP server address
  username: 'kevin@MY DOMAIN NAME'             # enter SMTP server username
  password: 'SECRET'             # enter SMTP server password

Web UI configuration for “lorawan-01.MY DOMAIN NAME”:

oauth:
ui:
canonical-url: “https://lorawan-01.MY DOMAIN NAME/oauth”
is:
base-url: “https://lorawan-01.MY DOMAIN NAME/api/v3”

HTTP server configuration

http:
cookie:
block-key: “secret” # generate 32 bytes (openssl rand -hex 32)
hash-key: “secret” # generate 64 bytes (openssl rand -hex 64)
metrics:
password: “cropwatch_metrics_HfHSQz4jYCnTRVhY” # choose a password
pprof:
password: “cropwatch_pprof_g4exjSfkmIzsLRQr” # choose a password

If using custom certificates:

tls:
source: file
root-ca: /run/secrets/ca.pem
certificate: /run/secrets/cert.pem
key: /run/secrets/key.pem

Let’s encrypt for “lorawan-01.MY DOMAIN NAME”

tls:

source: “acme”

acme:

dir: “/var/lib/acme”

email: “kevin@MY DOMAIN NAME”

hosts: [“lorawan-01.MY DOMAIN NAME”]

default-host: “lorawan-01.MY DOMAIN NAME”

If Gateway Server enabled, defaults for “lorawan-01.MY DOMAIN NAME”:

gs:
mqtt:
public-address: “lorawan-01.MY DOMAIN NAME:1882”
public-tls-address: “lorawan-01.MY DOMAIN NAME:8882”
mqtt-v2:
public-address: “lorawan-01.MY DOMAIN NAME:1881”
public-tls-address: “lorawan-01.MY DOMAIN NAME:8881”
forward:

Forward traffic to the Network Server in the cluster

• ‘cluster=SECRET/7’ # Enter your DevAddr range

Forward all traffic also to Packet Broker

• ‘packetbroker=SECRET/0’

Network Server configuration.

ns:
net-id: ‘SECRET’

If Gateway Configuration Server enabled, defaults for “lorawan-01.MY DOMAIN NAME”:

gcs:
basic-station:
default:
lns-uri: “wss://lorawan-01.MY DOMAIN NAME:8887”
the-things-gateway:
default:
mqtt-server: “mqtts://lorawan-01.MY DOMAIN NAME:8881”

Web UI configuration for “lorawan-01.MY DOMAIN NAME”:

console:
ui:
canonical-url: “https://lorawan-01.MY DOMAIN NAME/console”
account-url: “https://lorawan-01.MY DOMAIN NAME/console”
is:
base-url: “https://lorawan-01.MY DOMAIN NAME/api/v3”
gs:
base-url: “https://lorawan-01.MY DOMAIN NAME/api/v3”
gcs:
base-url: “https://lorawan-01.MY DOMAIN NAME/api/v3”
ns:
base-url: “https://lorawan-01.MY DOMAIN NAME/api/v3”
as:
base-url: “https://lorawan-01.MY DOMAIN NAME/api/v3”
js:
base-url: “https://lorawan-01.MY DOMAIN NAME/api/v3”
qrg:
base-url: “https://lorawan-01.MY DOMAIN NAME/api/v3”
edtc:
base-url: “https://lorawan-01.MY DOMAIN NAME/api/v3”
dcs:
base-url: “https://lorawan-01.MY DOMAIN NAME/api/v3”
oauth:
authorize-url: “https://lorawan-01.MY DOMAIN NAME/oauth/authorize”
token-url: “https://lorawan-01.MY DOMAIN NAME/oauth/token”
logout-url: “https://lorawan-01.MY DOMAIN NAME/oauth/logout”
client-id: “SECRET”
client-secret: “SECRET” # choose or generate a secret

If Application Server enabled, defaults for “lorawan-01.MY DOMAIN NAME”:

as:
mqtt:
public-address: “lorawan-01.MY DOMAIN NAME:1883”
public-tls-address: “lorawan-01.MY DOMAIN NAME:8883”
webhooks:
downlink:
public-address: “lorawan-01.MY DOMAIN NAME:1885/api/v3”

If Device Claiming Server enabled, defaults for “lorawan-01.MY DOMAIN NAME”:

dcs:
oauth:
authorize-url: “https://lorawan-01.MY DOMAIN NAME/oauth/authorize”
token-url: “https://lorawan-01.MY DOMAIN NAME/oauth/token”
logout-url: “https://lorawan-01.MY DOMAIN NAME/oauth/logout”
client-id: “device-claiming”
client-secret: “device-claiming” # choose or generate a secret
ui:
canonical-url: “https://lorawan-01.MY DOMAIN NAME/claim”
as:
base-url: “https://lorawan-01.MY DOMAIN NAME/api/v3”
dcs:
base-url: “https://lorawan-01.MY DOMAIN NAME/api/v3”
is:
base-url: “https://lorawan-01.MY DOMAIN NAME/api/v3”
ns:
base-url: “https://lorawan-01.MY DOMAIN NAME/api/v3”

Packet Broker Agent configuration

pba:

See https://packetbroker.net for available hosts

data-plane-address: ‘eu1.packetbroker.io:443’
net-id: ‘SECRET’
tenant-id: ‘SECRET’ # Leave empty if you own the NetID and you don’t use tenants
cluster-id: ‘’ # Unique identifier of your routing cluster
authentication-mode: ‘oauth2’
oauth2:
client-id: ‘SECRET’ # API key ID
client-secret: ‘SECRET’ # Secret API key value
forwarder:
enable: true
# generate 16 bytes (openssl rand -hex 16)
token-key: ‘SECRET’
home-network:
enable: true

In the device page, I see this error:

However, after searching for the error:

synthetic.error.network_error

I did find: Status of our TTG gateway is disconnected - name": "synthetic.error.network_error",
Though, I am sure I don’t have the ID duped, as I have only entered it once. Could this be because it is entered in my other instance still running on AWS?

I am wondering if there is a place where I can find detailed debug logs or something like this.

I am pretty sure the ports are all opened in my firewall though, it seems like they are not:

The ports in this image open the ports to the box running the reverse proxy, though I have also tried the one running TTS. I may change it back to see if there is another change.

The version I am currently running is: 3.25.1
everything is running on Ubuntu version 22.04.2 LTS
the reverse proxy, TTS, and one gateway are on the same network,
the 2nd gateway is on another network.

A little more interesting info:

I need to try an integration to see if it is sending data, and just not displaying it to me, or if it seems to be detecting the end, and not actually sending data between the TTS services

Any assistance would be greatly appreciated!
Thank you all very much,
-Kevinr

[Kevin192291]

Good news, I got everything figured out. It was NGINX’s configuration that was missing the packet TTL that was messing things up. Added that in, and everything is working perfectly now!