Setting up a Private Routing Environment


(Hylke Visser) #1

This topic can be used for comments and questions related to the guide for setting up The Things Network's routing services in a local or private environment.


How a newbie setup a private Lora network and to become an IoT operator?
Deploying a Private Routing Environment With Docker-Compose
Launching Production Environment
Private backend installation checkup & monitoring
Secure MQTT APIs now using Let's Encrypt certificates
TTN server build?
(Hylke Visser) #2

The guide was updated with the following:

  • Configured ttn-account-v2 account server
  • Added information about setting discovery server's CA certificate to ttnctl

#3

Hi @htdvisserm I followed the tutorial that you provided in the link above, and almost everything is going well, however now I get this error. I am runing the backend from a docker station, making use of your docker-compose file. I think that the IP address is not right in my case, is there a way to specify this IP address in the configuration?

discovery_1 | WARN Could not connect to Redis. Retrying... error=dial tcp: i/o timeout
networkserver_1 | WARN Could not connect to Redis. Retrying... error=dial tcp: i/o timeout
handler_1 | WARN Could not connect to Redis. Retrying... error=dial tcp: i/o timeout
discovery_1 | WARN Could not connect to Redis. Retrying... error=dial tcp: i/o timeout
networkserver_1 | WARN Could not connect to Redis. Retrying... error=dial tcp: i/o timeout
handler_1 | WARN Could not connect to Redis. Retrying... error=dial tcp: i/o timeout
router_1 | DEBUG Could not connect to gRPC server, reconnecting... Address=discovery:1900 error=dial tcp 172.20.0.4:1900: getsockopt: connection refused
broker_1 | DEBUG Could not connect to gRPC server, reconnecting... Address=discovery:1900 error=dial tcp 172.20.0.4:1900: getsockopt: connection refused
discovery_1 | WARN Could not connect to Redis. Retrying... error=dial tcp: i/o timeout
networkserver_1 | WARN Could not connect to Redis. Retrying... error=dial tcp: i/o timeout
handler_1 | WARN Could not connect to Redis. Retrying... error=dial tcp: i/o timeout

Thank you very much! :smiley:


(Hylke Visser) #4

@ndarkness: The guide only explained how to set everything up locally. In my next article I will will explain how to deploy using docker.


#5

Hi @htdvisser,

Thanks for the reply, would it be quite soon if I ask whether this is related to IP issues in docker?:innocent:


#6

Hi

I'm trying to start local ttn routing services on Windows.
I set discovery ttn.yml file as in your guide and try to get access token for router using:
ttn discovery authorize router my-router --config ./discovery/ttn.yml

I got warning:
WARN ttn: Failed to refresh public keys for token validation: Get file://discovery/key: unsupported protocol scheme "file"
and discovery server started wothout any information about access token.

Whats wrong ?


(Hylke Visser) #7

In the guide we use localhost everywhere. Docker uses different hostnames, so you'd need to have different certificates and different config. I'll explain everything soon :wink:

This sounds like you're using an old ttn version. Please check that you have the latest one.


#8

Thanks.
You are right. I've used old version.
Now I update source to v2.0.1 and I'm able to form access token for router, using ttn discovery command.
But when I try to start router I've got fatal memory error:
==========================================================

D:\Projects\lora\TheThingsNet>"D:\Projects\lora\TheThingsNet\src\github.com\TheThingsNetwork\ttn\ttn.exe" router  --config ".\router\ttn.yml"
Using config file: .\router\ttn.yml
  INFO Initializing The Things Network          Auth Servers=map[ttn-account-v2:https://account.thethingsnetwork.org] ComponentID=vitaly-router Description= Discovery Server Address=localhost:1900 Monitors=map[]
  INFO Initializing Router                      Announce=localhost:1901 Server=0.0.0.0:1901
  INFO Starting
 FATAL Stopping because of panic                panic=runtime error: invalid memory address or nil pointer dereference stack=goroutine 1 [running]:
github.com/TheThingsNetwork/ttn/cmd.Execute.func1()
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/cmd/root.go:110 +0x80
panic(0xce6920, 0xc042004090)
        D:/Go/src/runtime/panic.go:458 +0x251
crypto/x509.(*CertPool).AddCert(0x0, 0xc04235b200)
        D:/Go/src/crypto/x509/cert_pool.go:74 +0x4f
crypto/x509.(*CertPool).AppendCertsFromPEM(0x0, 0xc04235afdf, 0x0, 0x200, 0x45f)
        D:/Go/src/crypto/x509/cert_pool.go:113 +0x168
github.com/TheThingsNetwork/ttn/core/component.(*Component).initRoots(0xc0421ef0e0, 0x0, 0x0)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/core/component/auth.go:153 +0xd7
github.com/TheThingsNetwork/ttn/core/component.(*Component).(github.com/TheThingsNetwork/ttn/core/component.initRoots)-fm(0x0, 0x0)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/core/component/auth.go:32 +0x31
github.com/TheThingsNetwork/ttn/core/component.(*Component).InitAuth(0xc0421ef0e0, 0xc042171ce0, 0xc0423000f0)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/core/component/auth.go:39 +0x174
github.com/TheThingsNetwork/ttn/core/component.New(0x12f5b80, 0xc042371660, 0xde0064, 0x6, 0xc04238ef90, 0xe, 0xe, 0xdec4e4, 0x13)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/core/component/component.go:84 +0x522
github.com/TheThingsNetwork/ttn/cmd.glob..func15(0x13228a0, 0xc042360020, 0x0, 0x2)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/cmd/router.go:36 +0x1e6
github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra.(*Command).execute(0x13228a0, 0xc042305ea0, 0x2, 0x2, 0x13228a0, 0xc042305ea0)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra/command.go:636 +0x44a
github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1322680, 0x461208, 0xc042241e80, 0xc042241e90)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra/command.go:722 +0x36e
github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra.(*Command).Execute(0x1322680, 0xeb78e8, 0x4755bc)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra/command.go:681 +0x32
github.com/TheThingsNetwork/ttn/cmd.Execute()
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/cmd/root.go:116 +0x60
main.main()
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/main.go:23 +0x175

(Hylke Visser) #9

Didn't see that one coming...

Will fix that for you.


(Hylke Visser) #10

This should now be fixed in the develop branch, @vitalypru.


#11

Thank you.
Now I can start all ttn services.
But I faced with absolutely the same problem for lora-gateway-bridge


(Hylke Visser) #12

lora-gateway-bridge uses the same code to get the CA Roots from the system. We still have to update it to make it work with Windows.


#13

@htdvisser Could you tell me what does this error mean? Is the token that I am using completely wrong?

broker_1 | FATAL Could not initialize broker error=Failed to announce this component to TTN discovery: rpc error: code = 7 desc = permission denied: Discovery:Token issuer local is not allowed to make changes to the network settings: permission denied: Discovery:Token issuer local is not allowed to make changes to the network settings
ttnbackbone_broker_1 exited with code 1


(Hylke Visser) #14

Looks like your discovery server does not accept the "local" "master-auth-server". Make sure your discovery config contains the following:

discovery:
  master-auth-servers:
  - ttn-account-v2
  - local

Pretty sure this is in the guide as well. If not, let me know


#15

This is my current configuration for the discovery,

id: discovery
debug: true
key-dir: "./env/discovery/"
auth-servers:
ttn-account-v2: "https://account.thethingsnetwork.org"
#ttn-account-preview: "https://preview.account.thethingsnetwork.org"
local: "file:///root/env/discovery/server.pub"

tls : true
discovery:
redis-address: redis:6379
server-address: discovery
master-aut-servers:
- ttn-account-v2
- local

with that I get still have the error

discovery_1 | INFO ttn: Got public keys for token validation
discovery_1 | DEBUG Connected to gRPC server Address=localhost:1900
discovery_1 | FATAL Could not start client for gRPC proxy error=x509: certificate is valid for discovery, not localhost
ttnbackbone_discovery_1 exited with code 1


(Hylke Visser) #16

I suspect it's because the typo in master-aut-servers (there's a h missing)


#17

HI @htdvisser thanks again!

I have corrected it , but still the same issue

discovery_1      |   INFO ttn: Got public keys for token validation
discovery_1      |  DEBUG Connected to gRPC server                 Address=localhost:1900
discovery_1      |  FATAL Could not start client for gRPC proxy    error=x509: certificate is valid for discovery, not localhost
handler_1        |   WARN Could not connect to Redis. Retrying...  error=dial tcp [::1]:6379: getsockopt: connection refused

Edit: Why do I get the same error even if I run the discovery specifying the IP address of itself?

VirtualBox:~/TTNBackBone$ ./ttn-linux-amd64 discovery  --config="/home/lorabackbone/TTNBackBone/envLocal/discovery/conf.yml"
Using config file: /home/lorabackbone/TTNBackBone/envLocal/discovery/conf.yml
  INFO Initializing The Things Network          Auth Servers=map[local:file:///home/lorabackbone/TTNBackBone/envLocal/discovery/server.pub ttn-account-v2:https://account.thethingsnetwork.org] ComponentID=conf-discovery Description= Discovery Server Address=discover.thethingsnetwork.org:1900 Monitors=map[]
  INFO Initializing Discovery                   Database=localhost:6379/0 HTTP Proxy=0.0.0.0:8080 Server=0.0.0.0:1900
  INFO Starting                                
  INFO ttn: Got public keys for token validation
 DEBUG Connected to gRPC server                 Address=localhost:1900
 FATAL Could not start client for gRPC proxy    error=x509: certificate is valid for conf-discovery, not localhost

 
VirtualBox:~/TTNBackBone$ ./ttn-linux-amd64 discovery --discovery-address="172.20.0.4" --config="/home/lorabackbone/TTNBackBone/envLocal/discovery/conf.yml"
Using config file: /home/lorabackbone/TTNBackBone/envLocal/discovery/conf.yml
  INFO Initializing The Things Network          Auth Servers=map[ttn-account-v2:https://account.thethingsnetwork.org local:file:///home/lorabackbone/TTNBackBone/envLocal/discovery/server.pub] ComponentID=conf-discovery Description= Discovery Server Address=172.20.0.4 Monitors=map[]
  INFO Initializing Discovery                   Database=localhost:6379/0 HTTP Proxy=0.0.0.0:8080 Server=0.0.0.0:1900
  INFO Starting                                
  INFO ttn: Got public keys for token validation
 DEBUG Connected to gRPC server                 Address=localhost:1900
 FATAL Could not start client for gRPC proxy    error=x509: certificate is valid for conf-discovery, not localhost

#18

When I faced with the same problem, I found, that I miss to add "localhost" word in command line during discovery certificate creation.

$ ttn discovery gen-cert localhost --config ./discovery/ttn.yml


#19

Now I can't perform OTAA activation:

D:\Projects\lora\TheThingsNet>"D:\Projects\lora\TheThingsNet\src\github.com\TheThingsNetwork\ttn\ttn.exe" broker --config ".\broker\ttn.yml"
Using config file: .\broker\ttn.yml
INFO Initializing The Things Network Auth Servers=map[ttn-account-v2:https://account.thethingsnetwork.org] ComponentID=vitaly-broker Description= Discovery Server Address=localhost:1900 Monitors=map[]
INFO Initializing Broker Announce=localhost:1902 DeduplicationDelay=200 NetworkServer=localhost:1903 Server=0.0.0.0:1902
INFO Starting
DEBUG Connected to gRPC server Address=localhost:1900
INFO ttn: Got public keys for token validation
INFO ttn: Announced to TTN discovery
DEBUG Connected to gRPC server Address=localhost:1903
INFO Prefix found in Discovery DPrefix=26000000/20 NSPrefix=26000000/20
DEBUG Start stream CallerID=vitaly-handler CallerIP=127.0.0.1:55949 Method=/broker.Broker/Publish
DEBUG Start stream CallerID=vitaly-handler CallerIP=127.0.0.1:55949 Method=/broker.Broker/Subscribe
DEBUG grpc: Server.Serve failed to complete security handshake from "127.0.0.1:55961": EOF
DEBUG Start stream CallerID=vitaly-router CallerIP=127.0.0.1:55962 Method=/broker.Broker/Associate
DEBUG Connected to gRPC server Address=localhost:1904
WARN Could not dial handler for Activation AppEUI=70B3D57EF0000023 AppID=counter_app DevEUI=0088444D571864EB DevID=local_dev GatewayID=eui-a75c3affffe606e0 NumHandlers=2 error=x509: certificate signed by unknown authority (possibly bec
ause of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "serial:279521433737929252104238787983170638246")
DEBUG Connected to gRPC server Address=localhost:1904
DEBUG Forward Activation AppEUI=70B3D57EF0000023 AppID=counter_app DevEUI=0088444D571864EB DevID=local_dev GatewayID=eui-a75c3affffe606e0 HandlerID=vitaly-handler NumHandlers=2


#20

Thanks @vitalypru, the thing it's that I am using another "user" to sign the certificate, so I used

$ ttn discovery gen-cert conf-discovery --config ./discovery/ttn.yml

Did you run your private network in a docker container as well?