Help with "Failed to send webhook" Error

End Devices (Nodes)
1

Hi, I am getting a “Failed to send webhook” error that shows on the “Live Data” section of my end node.

This happens after an uplink of a 200 byte payload, I can see the values on the “Live Data” but nothing gets stored on my webhook. For my webhook I am using a custom template, with JSON format and no Downlink API key.

Does anyone know if this error relates to the webhook configuration? I have been able to store smaller payloads on this same webhook.

Here is the error metadata:

{
  "name": "as.webhook.fail",
  "time": "2021-11-10T07:35:16.575491631Z",
  "identifiers": [
    {
      "device_ids": {
        "device_id": "test-node-v2",
        "application_ids": {
          "application_id": "stm32wl-test-applications"
        }
      }
    },
    {
      "device_ids": {
        "device_id": "test-node-v2",
        "application_ids": {
          "application_id": "stm32wl-test-applications"
        },
        "dev_eui": "8080E1150500B78D",
        "join_eui": "0101010101010101",
        "dev_addr": "260DCAAE"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.ErrorDetails",
    "namespace": "pkg/applicationserver/io/web",
    "name": "request",
    "message_format": "request",
    "correlation_id": "83ba66983a43440e854c38ee4c448f37",
    "code": 14,
    "details": [
      {
        "@type": "type.googleapis.com/google.protobuf.Struct",
        "value": {
          "body": "Failed storing dump\n",
          "status_code": 500,
          "url": "http://ptsv2.com/t/zkudc-1620192826/post",
          "webhook_id": "test-app-webhook"
        }
      }
    ]
  },
  "correlation_ids": [
    "as:up:01FM4CBEJRG7EMRDTJ0C8MK39E",
    "gs:conn:01FM2REF0E24B6JQP555E0EZ5J",
    "gs:up:host:01FM2REF8XEK52VBNSQ5YFF88Y",
    "gs:uplink:01FM4CBEC5DH39H7372NR58VA5",
    "ns:uplink:01FM4CBEC7FTSC8XZDQ65ZKYWG",
    "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01FM4CBEC7X8530Z08NY5BFE6W",
    "rpc:/ttn.lorawan.v3.NsAs/HandleUplink:01FM4CBEJQN86JR5J7XN2C9CR1"
  ],
  "origin": "ip-10-102-14-148.ap-southeast-2.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_APPLICATION_TRAFFIC_READ",
      "RIGHT_APPLICATION_TRAFFIC_READ"
    ]
  },
  "unique_id": "01FM4CBFAZXVT7XAS8H0P54R1S"
}

Cheers,

2

Yes, I know. It does not relate to the configuration.

This is the third post in a matter of days on this so a forum search will yield more details but it appears the message appears when your end returns an error. You can test it with a range of HTTP clients or look at the server logs.

3

I ran a couple of tests with different payloads. Payloads up to 135 bytes were stored in the webhook without any problems. Anything above that gave me the same error.

4

It is YOUR end of the connection that’s the issue here - you need to fix your code - this is not a TTS issue - I can move over 200 bytes via a web hook if I absolutely need to.

5

We are using webhook to store LORA Raw data in our own database (for later analysis). that works fine. we are doing two of them independently for more data security.
We have a datagap on 2022-01-10 from 9-12 UTC on nearly all LORA nodes. How can I find out, if we had an issues, or if TTN webhook had an issue on that time. Is there a statistic avalaible to check if TTN webhook fails in the past?

6

There was a brief outage* following system update around that time see status pages for history - if you enabled data storage you might get lucky and be able to pull (some) data from there… (usually up to 24 hrs)

Suggest bookmark https://status.thethings.network/

(*) Many of us saw messages stop forwarding around 9:30/9:35 UKT, fixed arouund 11:30/12 UKT

7

I’d pay more for a better end server than have two.

If you are doing this because you are worried about webhooks from TTN, then if one goes down, the other will be down too.

So the really clever people like @Jeff-UK have a webhook & data storage as a backup.

However even DS was down yesterday.

If you are using TTN for real, then perhaps you should subscribe to the status notifications - direct to your phone for free.

8

I am doing two LORA DB to have a backup when domain server is doing server and at that time webhooks will run “in empty ends”. So I thought two different domain servers will help in this case.
I understand that when webhook is not running of course no data come in.
you suggest webhook + datastorage. I do understand technically webhook may fail but storage may still works.
I do understand datastorage is just a 2days “flight recording” storage = I see always the last two days. Is this correct?
**Question:**l is there an easy way to synchronize my LoRa DB with TTN storage to fill up gaps (maybe in php written)?
PS: I am happy about my LoRA DB: we had just the siutatin that our gateway started to send data from 6.1.1980. I learned this is GPS time, when GPS connection is not working. We now switched our import script to use keyword Received_At instead Time from upload message. And I needed to reimport data of the last 5 weeks :wink:

9

No - the docs give the definitive answer depending on which version of TTS you have.

Easy like open source code you can just download?

Or easy like buying?

Or easy like easy to write?

10

That is exactly what happened to me. My webhook had an error in it and all I could see in the console was “failed to send webhook”.

11

Hello,

Same issue, it says “message_format”: “operation timed out” or “message_format”: “request to {url} failed”, but after few minutes, it arrives to my server.

Thanks for your help

12

I’d suspect your server is slow to process and/or not returning a 200 OK. At the very least, it needs to return the 200.

It is preferable to have your own server in the same region and not on the cheapest host you can find. That said, it’s pretty hard to get a VPS for less than Linode offer and their kit is quick.