Maybe I can help you a bit further if you copy the error messages that you have here.
Sure, let’s try. Thanks!
From the browser console I can see a network package giving me an error when it tries to redirect me after logging in.
Request URL: https://redacted/console/oauth/callback?code=a_long_code&state=i0FSI5ummAkAUYPk
Status code: 403
This is the entire stack log from start to Token exchange refused:
stack_1 | INFO Setting up core component
stack_1 | WARN No cookie hash key configured, generated a random one hash_key=REDACTED namespace=web
stack_1 | WARN No cookie block key configured, generated a random one block_key=REDACTED namespace=web
stack_1 | DEBUG Loaded manifest.yaml mount=/assets namespace=web path=/srv/ttn-lorawan/public
stack_1 | DEBUG Serving static assets mount=/assets namespace=web path=/srv/ttn-lorawan/public
stack_1 | INFO Setting up Identity Server
stack_1 | INFO Setting up Gateway Server
stack_1 | INFO Setting up Network Server
stack_1 | INFO Setting up Application Server
stack_1 | INFO Setting up Join Server
stack_1 | INFO Setting up Console
stack_1 | INFO Setting up Gateway Configuration Server
stack_1 | DEBUG Loaded TLS certificate
stack_1 | INFO Setting up Device Template Converter
stack_1 | INFO Setting up QR Code Generator
stack_1 | INFO Setting up Packet Broker Agent
stack_1 | INFO Starting…
stack_1 | DEBUG Initializing gRPC server…
stack_1 | DEBUG Starting loopback connection
stack_1 | DEBUG ccResolverWrapper: sending update to cc: {[{in-process 0 }] } namespace=grpc
stack_1 | DEBUG Channel switches to new LB policy “pick_first” namespace=grpc
stack_1 | DEBUG Subchannel Connectivity change to CONNECTING namespace=grpc
stack_1 | DEBUG Setting up gRPC gateway
stack_1 | DEBUG pickfirstBalancer: HandleSubConnStateChange: 0xc000a1a690, {CONNECTING } namespace=grpc
stack_1 | DEBUG Channel Connectivity change to CONNECTING namespace=grpc
stack_1 | DEBUG Subchannel picks a new address “in-process” to connect namespace=grpc
stack_1 | DEBUG Exposed services namespace=grpc services=[ttn.lorawan.v3.AppAs ttn.lorawan.v3.ApplicationAccess ttn.lorawan.v3.ApplicationPackageRegistry ttn.lorawan.v3.ApplicationPubSubRegistry ttn.lorawan.v3.ApplicationRegistry ttn.lorawan.v3.ApplicationWebhookRegistry ttn.lorawan.v3.As ttn.lorawan.v3.AsEndDeviceRegistry ttn.lorawan.v3.AsJs ttn.lorawan.v3.AsNs ttn.lorawan.v3.ClientAccess ttn.lorawan.v3.ClientRegistry ttn.lorawan.v3.Configuration ttn.lorawan.v3.ContactInfoRegistry ttn.lorawan.v3.EndDeviceQRCodeGenerator ttn.lorawan.v3.EndDeviceRegistry ttn.lorawan.v3.EndDeviceRegistrySearch ttn.lorawan.v3.EndDeviceTemplateConverter ttn.lorawan.v3.EntityAccess ttn.lorawan.v3.EntityRegistrySearch ttn.lorawan.v3.Events ttn.lorawan.v3.GatewayAccess ttn.lorawan.v3.GatewayRegistry ttn.lorawan.v3.Gs ttn.lorawan.v3.GsNs ttn.lorawan.v3.GsPba ttn.lorawan.v3.GtwGs ttn.lorawan.v3.Js ttn.lorawan.v3.JsEndDeviceRegistry ttn.lorawan.v3.Ns ttn.lorawan.v3.NsEndDeviceRegistry ttn.lorawan.v3.NsGs ttn.lorawan.v3.NsJs ttn.lorawan.v3.NsPba ttn.lorawan.v3.OAuthAuthorizationRegistry ttn.lorawan.v3.OrganizationAccess ttn.lorawan.v3.OrganizationRegistry ttn.lorawan.v3.UserAccess ttn.lorawan.v3.UserInvitationRegistry ttn.lorawan.v3.UserRegistry]
stack_1 | DEBUG Initializing cluster…
stack_1 | WARN No cluster key configured, generated a random one key=REDACTED
stack_1 | DEBUG Initializing web server…
stack_1 | DEBUG Subchannel Connectivity change to READY namespace=grpc
stack_1 | DEBUG pickfirstBalancer: HandleSubConnStateChange: 0xc000a1a690, {READY } namespace=grpc
stack_1 | DEBUG Channel Connectivity change to READY namespace=grpc
stack_1 | DEBUG Initializing interop server…
stack_1 | DEBUG Starting gRPC server…
stack_1 | DEBUG Creating listener address=:1884
stack_1 | INFO Listening for connections address=:1884 namespace=grpc protocol=gRPC
stack_1 | DEBUG Creating listener address=:8884
stack_1 | DEBUG Loaded TLS certificate
stack_1 | INFO Listening for connections address=:8884 namespace=grpc protocol=gRPC/tls
stack_1 | DEBUG Started gRPC server
stack_1 | DEBUG Starting web server…
stack_1 | DEBUG Creating listener address=:1885
stack_1 | INFO Listening for connections address=:1885 namespace=web protocol=Web
stack_1 | DEBUG Creating listener address=:8885
stack_1 | DEBUG Loaded TLS certificate
stack_1 | INFO Listening for connections address=:8885 namespace=web protocol=Web/tls
stack_1 | DEBUG Started web server
stack_1 | DEBUG Starting interop server
stack_1 | DEBUG Creating listener address=:8886
stack_1 | DEBUG Loaded TLS certificate
stack_1 | INFO Listening for connections address=:8886 namespace=interop protocol=Interop/tls
stack_1 | DEBUG Started interop server
stack_1 | DEBUG Joining cluster…
stack_1 | DEBUG Joined cluster
stack_1 | DEBUG Starting tasks
stack_1 | DEBUG Started tasks
stack_1 | DEBUG Creating listener address=:8881
stack_1 | DEBUG Loaded TLS certificate
stack_1 | DEBUG Creating listener address=:1882
stack_1 | DEBUG Creating listener address=:8882
stack_1 | DEBUG Loaded TLS certificate
stack_1 | DEBUG Creating listener address=:1881
stack_1 | DEBUG Creating listener address=:1887
stack_1 | DEBUG Creating listener address=:8887
stack_1 | DEBUG Loaded TLS certificate
stack_1 | DEBUG Creating listener address=:1883
stack_1 | DEBUG Creating listener address=:8883
stack_1 | DEBUG Loaded TLS certificate
stack_1 | 2020/09/25 09:04:37 http: TLS handshake error from MY_IP: remote error: tls: unknown certificate
stack_1 | INFO Request handled duration=27.119µs method=GET namespace=web remote_addr=MY_IP request_id=01EK283NWXE8DVR8AEKS38BM8B response_size=45 status=308 url=https://MY_HOST_NAME/console
stack_1 | INFO Request handled duration=2.179706ms method=GET namespace=web remote_addr=MY_IP request_id=01EK283NZMA2KJN2A0D0J03JRH response_size=790 status=200 url=https://MY_HOST_NAME/console/
stack_1 | INFO Request handled duration=294.377552ms method=GET namespace=web remote_addr=MY_IP request_id=01EK283P5YBHFZE53MS8FMJ9J5 response_size=273513 status=200 url=https://MY_HOST_NAME/assets/console.c6ddc5f8af3f8c39f150.css
stack_1 | INFO Client error duration=801.97µs method=GET namespace=web remote_addr=MY_IP request_id=01EK283QMNZH4FC96J6KN13901 response_size=198 status=401 url=https://MY_HOST_NAME/console/api/auth/token
stack_1 | INFO Request handled duration=231.216µs method=GET namespace=web remote_addr=MY_IP request_id=01EK283QY8F1E31DRJVR4K70JK response_size=5004 status=200 url=https://MY_HOST_NAME/assets/logo.svg
stack_1 | INFO Request handled duration=452.974µs method=GET namespace=web remote_addr=MY_IP request_id=01EK283QY87G62PMDG9TXKFG9N response_size=0 status=302 url=https://MY_HOST_NAME/console/login/ttn-stack?next=/
stack_1 | INFO Request handled duration=222.014µs method=GET namespace=web remote_addr=MY_IP request_id=01EK283R2GE450CC42B7V002ZP response_size=0 status=302 url=https://MY_HOST_NAME/oauth/authorize?client_id=console&redirect_uri=%2Fconsole%2Foauth%2Fcallback&response_type=code&state=fQ_ugtpulr20LYET
stack_1 | INFO Request handled duration=1.108407ms method=GET namespace=web remote_addr=MY_IP request_id=01EK283R63F33VA1CTVJT1XKAK response_size=681 status=200 url=https://MY_HOST_NAME/oauth/login?n=%2Foauth%2Fauthorize%3Fclient_id%3Dconsole%26redirect_uri%3D%252Fconsole%252Foauth%252Fcallback%26response_type%3Dcode%26state%3DfQ_ugtpulr20LYET
stack_1 | INFO Request handled duration=593.912µs method=GET namespace=web remote_addr=MY_IP request_id=01EK283RECZQ85S12T8XJSRGCC response_size=82865 status=200 url=https://MY_HOST_NAME/assets/oauth.7974b415b4e1ab2ca5a4.css
stack_1 | INFO Request handled duration=1.647218362s method=GET namespace=web remote_addr=MY_IP request_id=01EK283REDP6YFY9ZMWSVHR3H1 response_size=1318285 status=200 url=https://MY_HOST_NAME/assets/oauth.ac5a77c09e306783f4cc.js
stack_1 | INFO Client error duration=684.629µs method=GET namespace=web remote_addr=MY_IP request_id=01EK283TKSAD6A9P94ZV253WBV response_size=177 status=401 url=https://MY_HOST_NAME/oauth/api/me
stack_1 | INFO Request handled duration=529.266µs method=GET namespace=web remote_addr=MY_IP request_id=01EK283TXAH8577NWQD66DYP7Z response_size=25520 status=200 url=https://MY_HOST_NAME/assets/source-sans-pro-v13-latin_latin-ext-600.117e12cdb861ed7356c805f6f515afbb.woff2
stack_1 | INFO Request handled duration=105.461865ms method=POST namespace=web remote_addr=MY_IP request_id=01EK28425T333E7HBWC9MXDNVJ response_size=0 status=204 url=https://MY_HOST_NAME/oauth/api/auth/login
stack_1 | INFO Request handled duration=67.869363ms method=GET namespace=web remote_addr=MY_IP request_id=01EK2842CRCMA52JDRWSQ3PRXZ response_size=0 status=302 url=https://MY_HOST_NAME/oauth/authorize?client_id=console&redirect_uri=%2Fconsole%2Foauth%2Fcallback&response_type=code&state=fQ_ugtpulr20LYET
stack_1 | WARN error=unauthorized_client, internal_error= get_client=client check failed, client_id=console namespace=identityserver
stack_1 | WARN OAuth error error=error:pkg/oauth:unauthorized_client (client is not authorized to request a token using this method) method=POST namespace=web remote_addr=127.0.0.1 request_id=01EK2842HTM7Z187V2B6293699 url=http://localhost:1885/oauth/token
stack_1 | INFO Client error duration=29.157257ms method=POST namespace=web remote_addr=127.0.0.1 request_id=01EK2842HTM7Z187V2B6293699 response_size=209 status=403 url=http://localhost:1885/oauth/token
stack_1 | INFO Client error duration=32.610047ms method=GET namespace=web remote_addr=MY_IP request_id=01EK2842HRBM3EN2QDQVJCE1M1 response_size=996 status=403 url=https://MY_HOST_NAME/console/oauth/callback?code=A_LONG_CODE&state=fQ_ugtpulr20LYET
stack_1 | INFO Client error duration=866.96µs method=GET namespace=web remote_addr=MY_IP request_id=01EK28432GCDSC5232X7Q77CCY response_size=198 status=401 url=https://MY_HOST_NAME/console/api/auth/token
stack_1 | INFO Request handled duration=679.644831ms method=GET namespace=web remote_addr=MY_IP request_id=01EK28436M2P6NRDWR6SXR0JCZ response_size=699912 status=200 url=https://MY_HOST_NAME/assets/ttn-console-bg.498252edc489187693a4e32162260925.png
While copying this, I noticed somehing that might give us a clue:
http: TLS handshake error from MY_IP: remote error: tls: unknown certificate
Does any of this ring a bell to you?
Indeed I recall to have a similar issue and the error message was a bit misleading. Stupid question what os are you using and did you open the ports on the firewall ?
I’m running this on an Amazon Linux 2 AMI and I opened the ports 80, 443, 1884, 8884, 1885, 8885, 1887, 8887 which are the ports that are used for management and such as documented here https://thethingsstack.io/reference/networking/.
Mmm okay I had it running on a local dedicated server. Did you sign the certificate ?
Yes, I’m using self signed certificates. I tried doing the ACME approach at first but it was giving me an error saying that the certificates couldn’t be found.
Well in this case I’m sorry but I cannot help you further in the end 
Something that I realized is that often the error messages provided are quite misleading. In my case it happened more than once that the issue was something totally different then the actual message was saying. Not a big help I know but yeah…
Good luck with your issue and keep us update if and how you managed to solve the issue
Update: I managed to make ACME work, so at least it will auto install the certificates where it needs to. I can now access via HTTPS correctly, but I still face the same Token Exchange Refused message.
Update 2: I DID IT! Turns out, you need to change most of the config options so that instead of localhost you point to your own domain name.
Example of a chunk of the changes I added to docker-compose.yml:
TTN_LW_CONSOLE_UI_AS_BASE_URL: https://mydomain:8885/api/v3
TTN_LW_CONSOLE_UI_IS_BASE_URL: https://mydomain:8885/api/v3
TTN_LW_OAUTH_SERVER_ADDRESS: https://mydomain:8885/oauth
TTN_LW_IS_OAUTH_UI_CANONICAL_URL: https://mydomain:8885/oauth
TTN_LW_IS_OAUTH_UI_IS_BASE_URL: https://mydomain:8885/api/v3
TTN_LW_APPLICATION_SERVER_GRPC_ADDRESS: mydomain:8884
TTN_LW_DEVICE_CLAIMING_SERVER_GRPC_ADDRESS: mydomain:8884
To know all the parameters you need to change just call ttn-lw-stack config | grep localhost and see what should be changed and what already works as is.
Also make sure the ports are open and accessible from the IP you are testing from AND from the server itself at least. Port 80 also needs to be open to Let’s Encrypt.
Also also I added the port 8885 to the command that creates the oauth client, so it ends up like this:
docker-compose run --rm stack is-db create-oauth-client --id console --name "Console" --owner admin --secret "console" --redirect-uri "https://mydomain:8885/console/oauth/callback" --redirect-uri "/console/oauth/callback" --logout-redirect-uri "https://mydomain:8885/console" --logout-redirect-uri "/console"
Not sure if this was really needed though, but it works for me so I’m not touching it ^^’.
2 Likes
Hi @christophmerscher if you don’t mind, I can use your help on how you got a self signed certificate working as part of your config? I am running a VM on a local network and using my own domain name? Eventually I will assign it to a public IP and use Let’s Encrypt automatically, but for now I would like some clear help on self signed certificate.
At the moment I have
Token Exchange Refused errors which as per configuration instructions is when you try to log in to The Things Stack Console, because an authorization request generated from within the container will not be redirected to port 1885 or 8885, where The Things Stack is listening.
Could you please help?
Hi @Tigere, I spent a bit of time facing this issue too but eventually got it to work. I was running TTS v3.9.4 in docker in a VM running Ubuntu 20.04, using self signed certificates in a made up domain name (ttn.local.com) that had not entries in any DNS.
To get mine to work I had to ensure the mapping of domain name to TTS stack docker network gateway ip address was put into both the VM machine /etc/hosts file AND in the TTS docker container /etc/hosts file (ie. 172.25.0.1 ttn.local.com).
Note to add the mapping to the tts stack docker container /etc/hosts file I found it easiest to add the extra-hosts property to the docker-compose.yml file so it adds the mapping to the docker /etc/hosts file automatically upon starting the container. Here’s a snippet of my docker-compose.yml file:
services:
…
stack:
…
extra_hosts:
- “ttn.local.com:172.25.0.1” # Ensure this IP address matches the docker network gateway address.
…
1 Like
Hi @TheIronNinja, the configuration options you’re specifying here are in our example ttn-lw-stack-docker.yml file -> https://thethingsstack.io/getting-started/installation/configuration/#example-configuration-files
If you set up The Things Stack according to the documentation, i.e with folder structure like
docker-compose.yml # defines Docker services for running {{% tts %}}
config/
└── stack/
└── ttn-lw-stack-docker.yml # configuration file for {{% tts %}}
And you use our example docker-compose.yml file, when you start the stack, the entrypoint line loads the ttn-lw-stack-docker.yml file with all of the configuration options:
stack:
# In production, replace 'latest' with tag from https://hub.docker.com/r/thethingsnetwork/lorawan-stack/tags
image: thethingsnetwork/lorawan-stack:latest
entrypoint: ttn-lw-stack -c /config/ttn-lw-stack-docker.yml
Glad you got this working. Running self signed certificates inside a Docker container with no DNS address is complicated! We’ve added instructions for using the machine’s local IP address, but adding the DNS records manually is a good solution.
@benolayinka, I can’t see any of @TheIronNinja’s config settings in the example ttn-lw-stack-docker.yml file. That file only contains TTN_LW_BLOB_LOCAL_DIRECTORY, TTN_LW_REDIS_ADDRESS, and TTN_LW_IS_DATABASE_URI. @TheIronNinja has specified 7 additional settings in their post, as well as adding port 8885 to the console oauth client registration.
Following up: I have managed to set up The Things Stack and log in to the console by following @TheIronNinja’s instructions above and using these steps for setting up https.
I am still having some issues with ttn-lw-cli (it’s trying to hit localhost and my certificate is for my machine’s static IP), but ttn-lw-stack config | grep localhost shows that there are some more config options using localhost which I can probably tweak to fix that.
Also of note is the following github issue which describes the IronNinja has mentioned: https://github.com/TheThingsNetwork/lorawan-stack/issues/1230
Referencing the open source configuration file here
TTN_LW_CONSOLE_UI_AS_BASE_URL: line 86
TTN_LW_CONSOLE_UI_IS_BASE_URL: line 80
TTN_LW_OAUTH_SERVER_ADDRESS: this is not a The Things Stack config option
TTN_LW_IS_OAUTH_UI_CANONICAL_URL: line 27
TTN_LW_IS_OAUTH_UI_IS_BASE_URL: line 29
TTN_LW_APPLICATION_SERVER_GRPC_ADDRESS: this is not a The Things Stack config option
TTN_LW_DEVICE_CLAIMING_SERVER_GRPC_ADDRESS: this is not a The Things Stack config option
1 Like
Thanks for clarifying, I didn’t realise some of those environment variables overlapped with the other settings in the file. I have now got ttn-lw-cli working properly after changing all the config variables in the github issue I linked. I’m going over my tts install process again to see which of these extra variables were needed. Once that’s done I’ll post the full process here so that (a) people googling this in the future can find it and (b) the install documentation can be updated as needed.
I’ve recorded the complete process for setting up TTS open source with a self-signed certificate here. It’s a bit long for a forum post, hence the pastebin link. Note that I’m using solution 1, replacing localhost with the static IP of my machine.
The key parts missing from the current TTS documentation are as follows:
- Uncomment the custom certs sections at the bottom of
docker-compose.yml(@benolayinka is adding this to the docs) - Uncomment the custom certs section near the top of
ttn-lw-stack-docker.ymland comment out the Let’s Encrypt section (@benolayinka is adding this to the docs) - Add additional environment variables to
docker-compose.yaml. In particular, I needed to add the following variables:
TTN_LW_APPLICATION_SERVER_GRPC_ADDRESS: 192.168.1.13:8884
TTN_LW_DEVICE_TEMPLATE_CONVERTER_GRPC_ADDRESS: 192.168.1.13:8884
TTN_LW_GATEWAY_SERVER_GRPC_ADDRESS: 192.168.1.13:8884
TTN_LW_IDENTITY_SERVER_GRPC_ADDRESS: 192.168.1.13:8884
TTN_LW_JOIN_SERVER_GRPC_ADDRESS: 192.168.1.13:8884
TTN_LW_NETWORK_SERVER_GRPC_ADDRESS: 192.168.1.13:8884
TTN_LW_OAUTH_SERVER_ADDRESS: https://192.168.1.13:8885/oauth
I’ve selected these variables by taking the full list from the relevant github issue and removing the variables which are already specified via config in ttn-lw-stack-docker.yml. Some (or all) of these variables could specified as config in ttn-lw-stack-docker.yml instead of as environment variables.
- Add port
8885tois.email.network.console-urlandis.email.network.identity-server-url. - Add port
8885to every url nested underconsole
Would it be possible to add these additional steps to the official configuration docs? I was not able to get my installation working until I found this thread and the linked github issue.
@htdvisser are these configuration options used somewhere? They aren’t documented. Is TTN_LW_NETWORK_SERVER a valid alias for TTN_LW_NS?
https://thethingsstack.io/reference/configuration/network-server/