Update from TTN - December 2020

Hi all! Hope you guys are doing well and are safe and healthy, wherever you are. Here in Amsterdam we’re just in a new lockdown that’s going to last for a while. It does give some extra time to focus on things that matter.

One of those things that matter is obviously making The Things Stack, known as V3, ready for the community to use. We’re already at stable version 3.10.4 and The Things Industries customers are already using The Things Stack for a while. It may sound strange, but making it available for the community is harder than for customers. That’s why it’s taking so long.

The main reason for this is scale. We’re seeing enormous increase of LoRaWAN packets globally over the past year. Here’s a chart of all LoRaWAN packets that Packet Broker received in the past 6 months (from TTN and TTI networks combined, globally, also traffic from other networks):

Screenshot 2020-12-17 at 15.19.24

(yes, we had a massive outage there)

That basically doubled. That means that in a few months time, we’ll be hitting more than a thousand messages per second.

With quite a complex system like The Things Stack, we need to do quite some fine tuning and monitoring. We followed the “first make it run, then make it run fast” development approach, but we need to make it fast before we can start handling this amount of traffic. The commercial The Things Stack Cloud is also steadily handling increased volumes of traffic, so we’re getting more and more confident and we know better and better what to optimize where and when.

The only big feature that we need in order for the community to start using The Things Network powered by The Things Stack, is the single sign-on with the community account. Since this is currently handled by the V2 Account Server, part of the V2 Stack, we decided to build a new component, called The Things ID. Your username and password stay the same. We’ll be using The Things ID for more new fancy things that I cannot announce yet.

All traffic of The Things Network will be available in the clusters powered by The Things Stack. We’re using Packet Broker for that. This means that we’ll be operating V2 and V3 side-by-side. We also have migration tools ready to migrate devices (and their security session) to the new environment. At some point, we’ll announce a sunset period at which we shutdown V2. When that happens, we’ll put forwarders in place to keep data feeding into Packet Broker, for gateways that are still connected to the old endpoints.

We start with community cluster eu1 that’s operated professionally by the The Things Industries on behalf of The Things Network Foundation. We’ll follow a similar deployment model as with V2 (i.e. US and Asia). I’m already having serious conversations with different communities to operate The Things Stack community clusters too, including UK, Eastern Europe, Australia, South Africa and India.

You can already find documentation on The Things Stack at https://thethingsindustries.com/docs. What’s particularly interesting is https://www.thethingsindustries.com/docs/getting-started/migrating/major-changes/ and the migration guides.

To keep you warm, here are some screenshots.

Login screen:

Screenshot 2020-12-17 at 14.41.00

Console landing page:

Screenshot 2020-12-17 at 15.47.58

Feel free to ask any questions, I’ll be around to answer them.

23 Likes

Great news Johan.

Some questions since you asked for them. Sorry to hear you’re in lockdown but great to see the productive redirection. Look after yourselves first.

  1. Will there be more monitoring for the TTN stack with this upgrade? v2 has had a variety of interesting failure modes (inter-region routing, stateful (stale) gateway connections (for Semtech forwarders at least), and others I’m less aware of.
    1a. Do you want some community initiatives on this?
  2. Do you see net coverage increasing with TTN being connected to PacketBroker? Similarly is non-TTN traffic heard from our gateways being sent to TTI/others e.g. loriot.io /KPN/etc now? (This would be a great thing for spectral efficiency).
  3. Any plans for >8 channels for TTN being defined? We’re now seeing 16 and 64 channel gateways in the market; at some point local TTN bands will saturate even with the Fair Usage Policy.
    3a. Is the TTN FUP enforcement on the horizon now?
  4. How frequently do you see TTN version updates happening post-upgrade? (super-frequent e.g. push-on-green would be my preference, stagnating is not safer/better measured over the long term).
    4a. Can we do anything to help the tests for this? or participate in canary instance testng?

Thanks again.

/Bruce

2 Likes

The Things Stack is written from the ground up. There’s no V2 code in there, only many lessons learned. That includes the way we connect gateways and how traffic flows between clusters (via Packet Broker).

As for monitoring, The Things Stack comes with an event system that allows you to monitor hundreds of things, including gateway connect/disconnect, where messages come from (with Packet Broker metadata), downlink scheduling, MAC command scheduling, etc. We are working on visualizing that better in the Console, but otherwise it’s available via API and CLI.

We’re currently not offloading traffic outside The Things Network ecosystem or ingesting traffic from other networks. So while community networks and commercial managed private networks are interconnected because we can define the rules, other networks are not (yet).

The basic rules are written in 2015 (see GitHub - TheThingsNetwork/Manifesto: The Things Network Manifesto). If other networks come and want to exchange traffic with TTN, and contribute back in an equal and open manner, then that’s completely fine. We are opening up Packet Broker (and TTN traffic exchange) with users of the open source The Things Stack and ChirpStack, because there are plenty of good reasons to manage and operate your own LNS, but still want to contribute to and use the coverage of the community network.

We’ve had many discussions with other networks, but if one has a per-gateway pricing model (Loriot), you better protect your smart city business with only offering your network services to gateways that are being paid for. TTN is pretty open here, but it’s still mostly seen as a threat to other business models, unfortunately.

Besides spectral efficiency, it’s also good for battery life.

We are going to support multiple frequency plans so you can combine them on 16 or 64 channel gateways. FUP is not being enforced, but we will start monitoring it and showing a feedback loop to users.

We usually publish releases every 1 to 3 weeks, see Releases · TheThingsNetwork/lorawan-stack · GitHub. We are currently working on continuous delivery to staging clusters. Indeed, we might provide a public beta cluster to the community, so that integration tests can be run there.

8 Likes

Hi Johan
The truth is that the migration to version 3 of the stack is fabulous, I think it is the most anticipated news by all.
The question is if it is already available to start the migration and if it is necessary to use any special access for the new console or will it just appear instead of the old V2.0 console?
Thank you very much in advance, we are looking forward to testing Class C devices at TTN !!!
I wish all colleagues have a very happy and prosperous year 2021, with health, work and love, and that you can enjoy it as a family.

Hugo Müller - TTN INITIATOR PARANÁ ENTRE RÍOS ARGENTINA

Thanks for your kind words Hugo, we wish you the very same for 2021.

V3 is not available for migration yet; we’ll be doing some extensive testing beforehand.

It won’t appear instead of the old V2 Console. We keep the two versions running side-by-side, and Packet Broker will make sure that gateway coverage is available in the two environments.

1 Like

Cheers Johan, I love all these details. I too can’t wait for the new stack, especially the BasicStation support.

Great! I would put my local (accessible!) TTN gateways on that cluster, and I’m sure others would, I would love TTN to stay at HEAD on a permanent basis. Real-world canarying e.g. as part of the rollout process would let you move faster/more confidently and get a better overall result. I’m not sure you were going that far (sounded like pre-prod integration) but please consider a production canary cluster?

I’m also wondering what’s on your wishlist from the TTN community for the next year? What should Sinterklaas bring you next year? :slight_smile:

Something like:

  1. Monitoring/alerting off the new stats? status.thethings.network is a great start, but an external community-maintained dashboard and alerting would be pretty cool (and maybe even cut down on the “my GW looks offline” threads). I’m not asking for more from your team, to be clear.
  2. Other things? There hasn’t been a high level of external contributions to the stack (and TTNv2 being in use but deprecated hasn’t helped), but equally we don’t know what gaps you see that need filling? That’s not just a stack code thing, the build, rollout, monitoring and testing infra too perhaps?

Happy New Year,
Bruce

You are (part of) the community, feel free to start working on/discussing this.

1 Like

We are definitely considering exactly that indeed. We are already building snapshots on each merge to master; we just don’t have the canary cluster yet and the automated update. But it’s the plan.

I have little to ask after all the patience waiting for V3. When it’s there, constructive feedback is very welcome, in the form of issues that can be filed at Issues · TheThingsNetwork/lorawan-stack · GitHub.

Besides that indeed some community driven monitoring solution would be helpful. It’s always better to have things tested externally, to avoid bias and black spots.

Also, language SDKs would be welcome; Python, Node, Typescript, Java, Rust among others. We have a few issues to close before this becomes easier for SDK developers.

4 Likes

Wow, all this sounds great, especially a community for South Africa sounds great. I would like to see a server also deployed in the future. I am also working on creating a server to handle South Africa, and the frequencies. As other servers either do not comply or are out of reach…
Thanks for the great work at The Things Network