TTN message "Failed to send webhook"

Hello Everyone,

For almost three weeks, I am observing something wrong with TTN. The TTN actually received the message from the sensors and it is unable to send it to our webhook. Did anyone also face the same issue? I get the following error.

ttn-error-2

Any leads would be highly appreciated. Thanks.

Kind regards,
Imran Khan

1 Like

What kind of webhook does it use? What does the configuration of your web hook look like?

Hello @bertrik,

The Webhook works normally fine. This abnormal behaviour from TTN appears just for one or two hours and then it starts working fine. I don’t think that I have an issue with Webhook settings or our backend API.
I did not explain my problem in the first place. The main issue that I am having now is that for a certain period of time I get this Failed to send the webhook message in the TTN and at this time I also don’t get any data at my endpoint API. After some hours, the TTN starts again sending data to Webhook and I also receive data at my endpoint API.

Kind regards,
Imran Khan

Hi Imran, what does the error metadata tell you? Click on one of the error messages to expand and see more… you may get a clue. I’m busy migrating GW’s and nodes to V3 and missed fact one of my early migrations was throwing a similar webhook error. When I checked I saw it was still linking to a temp server set up by Nick @descartes to support the Things Summer Academy…now long since switched off. I simply deleted the errant webhook. You may find similar clues… a malformed/corrupt url, dead service, changed connection/service no longer correctly formatted or whatever… worth a peep. If you don’t understand why you see capture as text (not a picture) and post here for others to assist where poss.

It would be super useful to have @bertrik’s question answered:

What kind of WebHook is it?

Hi @descartes. It is a simple PHP API. Below is the setting screenshot.

ttn-error-3

Thanks

Hello @Jeff-UK,

Below is the error metadata.

{
  "name": "as.webhook.fail",
  "time": "2021-10-26T09:51:53.256039533Z",
  "identifiers": [
    {
      "device_ids": {
        "device_id": "dragino-lse01-23",
        "application_ids": {
          "application_id": "dragino-soil-lse01"
        }
      }
    },
    {
      "device_ids": {
        "device_id": "dragino-lse01-23",
        "application_ids": {
          "application_id": "dragino-soil-lse01"
        },
        "dev_eui": "A84041CAA182E3BC",
        "join_eui": "A000000000000100",
        "dev_addr": "260B3462"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.ErrorDetails",
    "namespace": "pkg/applicationserver/io/web",
    "name": "request",
    "message_format": "request",
    "correlation_id": "a0a2a68d87ea4000b76af15f90e2a06a",
    "code": 14,
    "details": [
      {
        "@type": "type.googleapis.com/google.protobuf.Struct",
        "value": {
          "body": "<!doctype html>\n<html class=\"theme-light\">\n<!--\nError: Object of class stdClass could not be converted to string in file /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php on line 332\n\n#0 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemManager.php(398): Illuminate\\Filesystem\\FilesystemAdapter-&gt;prepend()\n#1 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(261): Illuminate\\Filesystem\\FilesystemManager-&gt;__call()\n#2 /var/www/gcdb/app/Http/Controllers/SensorDataController.php(49): Illuminate\\Support\\Facades\\Facade::__callStatic()\n#3 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): App\\Http\\Controllers\\SensorDataController-&gt;getData()\n#4 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(45): Illuminate\\Routing\\Controller-&gt;callAction()\n#5 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Routing/Route.php(262): Illuminate\\Routing\\Controlle",
          "status_code": 500,
          "url": "https://gcdb.hait.it/api/sensorData",
          "webhook_id": "push-to-db"
        }
      }
    ]
  },
  "correlation_ids": [
    "as:up:01FJY06TPP85HNB39CG3T6FFCJ",
    "ns:uplink:01FJY06TFG46PHAKFPTXRFVKR9",
    "pba:conn:up:01FJV6TBFG55W13SQ81J0PN1R0",
    "pba:uplink:01FJY06TEYXJ565VM1KFS7FMMA",
    "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01FJY06TFG5WQC2W5GK5ZSEC37",
    "rpc:/ttn.lorawan.v3.NsAs/HandleUplink:01FJY06TPNKGXYAYYWKK5P2YTZ"
  ],
  "origin": "ip-10-100-13-103.eu-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_APPLICATION_TRAFFIC_READ",
      "RIGHT_APPLICATION_TRAFFIC_READ"
    ]
  },
  "unique_id": "01FJY06TX8Y4T0QBAD09GRFDT3"
}

Kind regards,
Imran Khan

Thanks.

Why are you including both a Downlink API key and an Authorization header with a TTS key? The first is so you can do Downlinks - the other headers are only for your convenience. Are you using the key to do other things? If so, nice trick. I’m going to use that as it saves embedding the key in the PHP settings!

I think the simplest thing to do will be to run another webhook in parallel that doesn’t need the extras. Perhaps if you use this:

as is - it literally just logs the incoming messages to tab files - and has been well exercised by the Summer Academy students - it will tell us if there is an issue with the Downlink API Key & additional headers setup. Which will be all on the TTS side as it is literally fire & forget.

1 Like

As always, thank you so much @descartes for your valuable suggestion and guide. Actually, I just analyzed the metadata and found this error is because of my API.

This is just for validation/authorization.

As you have experience in this field, I am wondering, why the same API works for sometimes and why it does not work for sometimes? Does the TTN send different metadata for the same kind of sensors in the same application?

Kind regards,
Imran Khan

1 Like

Which error?

Was your API returning an error code - not something I’ve tried but again, will give that a go - I wasn’t aware that the TTS waited around for a response!

As far as I know it sends exactly what we see in the console - so it’s whatever comes up after the payload formatter.

I guess this is an error, which comes from the Laravel framework.

`"body": "<!doctype html>\n<html class=\"theme-light\">\n<!--\nError: Object of class stdClass could not be converted to string in file /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php on line 332\n\n#0 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemManager.php(398): Illuminate\\Filesystem\\FilesystemAdapter-&gt;prepend()\n#1 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(261): Illuminate\\Filesystem\\FilesystemManager-&gt;__call()\n#2 /var/www/gcdb/app/Http/Controllers/SensorDataController.php(49): Illuminate\\Support\\Facades\\Facade::__callStatic()\n#3 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): App\\Http\\Controllers\\SensorDataController-&gt;getData()\n#4 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(45): Illuminate\\Routing\\Controller-&gt;callAction()\n#5 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Routing/Route.php(262): Illuminate\\Routing\\Controlle",`
[/quote]

I was more thinking the HTTP status code - like an error 500.

I would say that loading a whole Laravel framework to capture an incoming uplink is a sledge hammer to crack a nut and not very scaleable.

Actually, we have a whole web application running on the Laravel Framework and this TTN WebHook API is only one small part.

I’m not saying don’t have a web application - I’m saying that the load on your server from an incoming uplink will be variable and that using a web application framework to parse some JSON and put it in to a database seems like asking for scaling issues.

Not that I’ve been coding Personal Home Page since Rasmus bought it out at the same time Håkon created CSS but just before Brendan bought us LiveScript, just at the time Tim drafted a standard for his HTML thing. Not sure quite how it got so out of hand since, I blame JQuery, it started a trend.

1 Like

If you’re building a larger application, it’s totally fine to use a framework like Laravel/Rails/Django/… and implement webhook receiving logic into that larger application.

I think the conclusion of this discussion is that this is not an error in The Things Stack, but an error coming from @LoRa_AuH’s application (formatted for readability):

<!doctype html>
<html class="theme-light">
<!--
Error: Object of class stdClass could not be converted to string in file /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php on line 332

#0 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemManager.php(398): Illuminate\Filesystem\FilesystemAdapter-&gt;prepend()
#1 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(261): Illuminate\Filesystem\FilesystemManager-&gt;__call()
#2 /var/www/gcdb/app/Http/Controllers/SensorDataController.php(49): Illuminate\Support\Facades\Facade::__callStatic()
#3 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): App\Http\Controllers\SensorDataController-&gt;getData()
#4 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(45): Illuminate\Routing\Controller-&gt;callAction()
#5 /var/www/gcdb/vendor/laravel/framework/src/Illuminate/Routing/Route.php(262): Illuminate\Routing\Controlle...

So this is not something we can fix on the TTN side, but should be fixed in your code.

1 Like

Sure, you CAN.

But should you?

Would you?

And the OP has already confirmed it was an issue at his end …

is this problem solved because i am also having the same problem?

It isn’t a TTN issue - it was due to a server side error - please read the thread for details.

1 Like

How did you solved this problem? because I am also suffering this problem , at ThingSpeak in csv file entries are show with timestamp value but not any data. here I am explaining how do integration

  1. create new channel
  2. Copy the Write API Keys
  3. The Thingstack open
    TTN Integration–>Enter Channel ID, paste WRite APIKeys and create

After this i am also suffering from problem . Please help . Please give solution step by step . Thankyou so much
and I am senting an integer value from LoRaWAN node to multigateway channel my data is reached at ThingStack but from here not reached to Thingspeak

What is this payload? Is it MAC payload? I send the packet from LoRa node to gateway after that packet goes to TTN server. What is this packet carrying? Please explain each and every part of the packet. and how retrieve this packet?
Thankyou