TTN GATEWAY central

I would imagine that enabling debug logging has a similar effect as the proposed beta patch (halving the baudrate) in effect slowing communication / execution down sufficiently to mitigate the timing issues with the internal clock?

1 Like

The firmware mentioned in this post also gives me the same issue as before … I have confirmed that gateway has been updated through SD card, is running fw version v1.0.1-facdef23 (2018-02-14T08:16:22Z), and I have disabled OTA updates.

I also have a connector with etched number 1636D4 if that matters.

UPDATE 1: after dozen of reboots during 30+ minutes, gateway finally connected to the broker for the first time. Its been connected for a few minutes now which is an improvement compared to before where it rebooted every minute before even connecting to the broker.

UPDATE 2: after half a day the gateway is again rebooting. Sigh.

I confirm that my Malfunctioning gateway has a 1636D4 socket; my gateway seem no long able to get past the power led solid with the second led flashing rapidly; and its not visible on the WiFi or wired network or as an AP?

Also tried flash new firmware without success :’(

it would explain why its so hard to track down!? sounds like a timing issue that is resolved by logging; a missing lock/mutex perhaps ? wonder if its worth faking a few on the task to try and identify the area ie

run with only the lora see it reboots
repeat above adding wifi
repeat above adding ethernet

you get the picture; @arjanvanb thanks for the info; just hope many gateway can be fixed :’(

I also tried the new firmware. No success. Keeps rebooting.
I also have a 1636D4 connector…

Most likely, the people at TWTG/Microchip did only test the gateway software with debug logging enabled. Understandable when you are developing the software but any professional in software ought to know that you must finally test the software as delivered to the customer. Putting debug logging in dramatically changes the timing or performance so things suddenly stop to work when those delays due to logging (especially to a serial terminal) are removed. As they had two years to complete the software due to hiccups in the hardware manufacturing one wonders why these issues have not been found earlier.
Another token to the immaturity of the whole is the fact that the gateway has to rebooted every 24 hours. Could you imagine that to be done on the average Linux server? Embedded software should be designed to run for years, not hours!

3 Likes

I agree, for mission critical, you can’t rely on volunteer based GW because as you said, if user remove or power off, you are stuck.
But you can just put you own GW is safe place, connected to TTN, the backend is pretty fine stable and you can rely on.That’s what I’m doing for industrial customers, never complains and if so we can arrange a local backend of another one, then they look at the price and stay on TTN and even add reliable GW to the community. So I don’t agree saying TTN is only for hobbyists, all network stack and backend is far better than commercial ones and V3 is coming will be may be the best ever seen.
And all of it for free, just unbeatable

2 Likes

@BoRRoZ, I argued that way as well. @GrahameH made that point, not me.

And as I said, I am impressed by the central infrastructure too, @Charles!

1 Like

Hi,

Just received my Gateway, and tried to activate it via https://ttn.fyi/activate

The Gateway properly connects to my WiFi, and I can see the status via http:///info

However, it keeps rebooting, and it never connects to The Things Console.

What can I do to fix this?

Thanks,
Yoeri

Upgraded Firmware from v1.0.0-917719b9 (2017-06-26T17:59:33Z) to v1.0.1-facdef23 (2018-02-14T08:16:22Z). Does not seem to have fixed my reboot loop. Still not connecting to Console.

Status Update:
Updating my Gateway via USB did at least take me to the point where the gateway forwarded the first packet to the TTN network. :+1:
I needed to disable the auto update in my TTN account, as i was stuck in an update loop :wink:
But it is far away from stable. I am not sure if it is hardware or software issue. Every time i slightly adjust the position of the LoRa card, it seems to be stable for some minutes but then the next reboot occures.

Additionally the packets from my TTN node are transmitted according to debug log, but i dont see the device on the activation map. But that is a topic for an other thread here i think.

Michael

Does it say ‘Gateway Card: 868Mhz’ (or your regional frequency) in the info output?

If it doesn’t and it says ‘ND’ instead of a frequency, you may want to try to push the Lora circuitboard in a bit firmer.

Earlier this week I fired up my gateway and it kept on looping as well, pushing the board a bit fixed it for me. It has been completely stable since.

My only concern now is that the gateway struggles to outperform my 2.4GHz wifi signal when it comes to range :expressionless:

But note that the info is updated every second or so. After booting, it will always say “ND” (“not detected”?) for a while.

People, let’s try to keep it a bit professional here. We’re not a bunch of children, so let’s not make this topic about personal attacks and name-calling. Based on the comments in this topic that have been posted over the past 24 hours, I think we should all take a step back and focus on what’s actually going on.

Some people are experiencing reboots of the gateway. This sucks and it doesn’t meet the quality that many of you have come to expect from TTN based on your experience with us over the past years. The @TTP team is working hard on a solution, but that takes time. Based on research by @arjanvanb, and the @TTP team, the problems have been traced back to a clock/timing issue. A patch by @arjanvanb seemed to solve the problem for some gateways, but doesn’t completely fix the problem. The @TTP team is working on a patch that should fix the problem permanently. This takes time and requires long-running tests to make sure that nothing else breaks. As soon as (intermediate) patches can be safely released, they will be pushed to the beta update channel.

Regarding @Ajj’s message: The gateway has undergone extensive testing (with production firmware, not just debug firmware), and there are currently hundreds of The Things Gateways online that do not have any reboot issues. The 24-hour restarts will be disabled as soon as we’re sure that the firmware is stable completely stable. Until then, these restarts provide a way to roll out critical firmware updates.

I can understand @GrahameH’s frustration, but I’d like to address a couple of his claims that I think are a bit unfair. First of all, @GrahameH seems to generalize the issues with his gateway to failure of the entire project/team. Looking at the progress we’ve made since I joined the core team full-time two years ago, I think we’ve built a pretty awesome project. We’ve brought together over 30,000 people worldwide in our communities, of which 750 attended our conference in Amsterdam. Our free public community network is successfully routing a couple of million messages per day, both for hobbyists and commercial users. So I strongly disagree with the claim that “this product and network has no commercial ability”.

There have also been a couple of messages about “mission critical communication”. I’d like to clarify that neither TTN nor LoRa(WAN) or actually any ISM radio technology can safely be used for truly mission critical communication. ISM spectrum is the wild west of radio, nobody can guarantee a successful transmission. LoRa(WAN) is a young technology, and though it works pretty well, it’s indeed not yet as mature as GPRS/3G. TTN is a community project, don’t rely on people you don’t know to keep their gateways online. TTN’s gateway firmware is still under development, and it’s indeed not completely stable yet. Future firmware updates will fix that. It’s all open source, so feel free to help out.

Sending repeated messages/emails/forum posts about your gateway won’t get it fixed faster. So consider letting the @TTP team work on a solution instead of expecting them to write replies to your messages.

Again, let’s try to keep the criticism constructive. Try to help find solutions, not just highlight problems. Try not to offend people and try not to feel offended. And let’s focus on what we’re all here for: building this thing together.

13 Likes

Thank you, Yes I’m frustrated because IMHO while everyone is focused on this reboot issue and it seem noting is getting done with regards the issue I’m seeing, while I do understand the team has/is working hard.

I DO hope and trust the team will get round to the type of issue I’m having, but I’m at a lost to understand how I can resolve it as the gateway won’t update or show as a AP

If I have offended anyone here then please accept my sincere apologies, it was not intended; I have been trying to keep to facts and experience with my feelings as IMHO

With regards to the mission critical statement, this was a poor attempt and a personal opinion at possible failure during testing/quality control and/or the mindset of the overall project which I unreservedly apologise for if this caused offence.

Please, please could someone acknowledge the issue I’m seeing and provide some guidance.

4 Likes

I think they’re indeed focusing on the reboot issue, which seems to affect more people and therefore has higher priority. That doesn’t mean that we don’t care about your issue, but giving one problem the full attention will get it solved quicker than dividing time over two (unrelated) issues.

Your issue is tracked as TheThingsProducts/gateway#5. The last message in that issue (from you) says “after many attempts and reboots the gateway is now showing up as an AP and I’m able to connect; and the gateway now looks to have successfully updated it’s firmware” […] “however the #1 is still present” (where #1 is the reboot issue). If you are again seeing the AP issue, please comment on Github with a detailed message of what’s happening from the moment you plug in the gateway, so that the @TTP team hopefully has enough information to look into it when they have time. It would also be helpful to see what happens if you use an ethernet cable to connect the gateway. Please write all of that in the Github issue so that we have everything in one place, instead of information being spread across multiple places.

1 Like

Ok, have done, and described as possibly intermittent; will leave for extended period to see if that changes anything

To keep track of issues I propose to register issues at the github repo. It is the central location and will help to keep track and overview of everything.

2 Likes

Good post!
But if you talk about hundreds of gateways running fine the failure rate seems quite high, maybe even 10% if you count the issues reported here.

Yes, it first says ND, but changes to 868Mhz before rebooting.