Downlink via Blynk to HTTP Integration

[TobiasPe]

Hey Community,
i want to initiate downlinks with my Blynk app.

I set up an Application in TTN with an HTTP Integration. Data forwarding from TTN to Blynk works fine. Now i want to go the reverse way. Putting the Downlink-Link in Postman (POST method and data in JSON format in the body) gives me proper downlink initiations (i know, only 10 per day). But doing the same in Blynk results in no action in the TTN console. For testing i made both (postman and blynk) send data to requestbin, to see what traffic and data actually is arriving:

Postman (successful downlinks):

Blynk (not showing up in the ttn console at all):

any ideas why this is not working? Changing the content-type in blynk doesn’t help either. Requestbin says in the latter case: content-type text/plain; charset=UTF-8

[arjanvanb (Arjan)]

Did you also add some ?key=ttn-account-v2.dummy-secret part to the URL of requestbin.com? Maybe Blynk is doing some funny encoding of the question mark, or of the parameter value.

I don’t know Blynk, but how did you set it up? Can you test without a User-Agent header in Postman?

@TobiasPe

Requestbin says in the latter case: content-type text/plain; charset=UTF-8

So you set it to text/plain and Blynk changes that into text/plain; charset=UTF-8?

I wonder if Blynk always does that, or if it sees some funny characters that are not plain ASCII, and adds that encoding instruction. Does copying the JSON content into some editor show any funny characters, or differences in line endings? Maybe copy the JSON from Blynk into Postman and POST that? By the way: application/json should be fine of course. Unless that JSON indeed includes some funny characters, like non-breaking whitespace.

(Aside: https://integrations.thethingsnetwork.org uses a Let’s Encrypt server certificate, with I assume Blynk should support.)

[TobiasPe]

@arjanvanb

Did you also add some ?key=ttn-account-v2.dummy-secret part to the URL of requestbin.com?

i can’t do that. modifying the url. in. requestbin is not possible. but doing something like this: https://en9iip5qks.x.pipedream.net?

ended up in → …/?

i think blynk alters the url in some way.

@arjanvanb

Does copying the JSON content into some editor show any funny characters, or differences in line endings?

no, this remains unchanged

so i tried so send an http post request with the esp8266wifi library. the problem there is that my esp8266 doesn’t support https and the ttn-http-integration doesn’t support http. correct?

anyways: this doesn’t work either:

WiFiClient client;
    HTTPClient http;
    String ttn_downlink_url = "https://integrations.thethingsnetwork.org/ttn-eu/api/v2/down/tobipeloratest/forward_gps_data?key=ttn-account-v2.dummy";
    http.begin(ttn_downlink_url);
    http.addHeader("Content-Type", "application/json");
    http.POST("{\"dev_id\":\"downlink_tester_otaa\",\"payload_raw\": \"VG9iaWFz\",\"port\": 1,\"confirmed\": false,\"schedule\": \"replace\"}");
    String api_answer = http.getString();
    Serial.println(api_answer);
    http.end();

[arjanvanb (Arjan)]

@TobiasPe

i can’t do that. modifying the url. in. requestbin is not possible. but doing something like this: https://en9iip5qks.x.pipedream.net?

ended up in → …/?

what do. you. mean “i can’t do that”?

It’s about adding the URL path and (above all) the query ?key=... to the URL that Blynk uses to make the request. Without Blynk, that works fine for me; a POST to https://eno3w4uq0suaq.x.pipedream.net/ttn-eu/api/v2/down/my-app-id/my-process-id?key=ttn-account-v2.secret nicely shows:

fetch('https://eno3w4uq0suaq.x.pipedream.net/ttn-eu/api/v2/down/my-app-id/my-process-id?key=ttn-account-v2.secret', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({someKey: 'someValue'}),
})

@TobiasPe

the ttn-http-integration doesn’t support http. correct?

I’ve never tried, but it makes sense, this being 2020. But if the ESP8266 WiFi library does not support it, then the following should surely use an http:// URL, not https://:

@TobiasPe

String ttn_downlink_url = "https://integration ...

Don’t forget to test without a User-Agent in Postman!

[TobiasPe]

@arjanvanb

It’s about adding the URL path and (above all) the query ?key=..

Done that and it works, but only with RequestBin:

TTN-HTTP-Integration is still not able to ‘read’ my request through Blynk.

@arjanvanb

success!

Ok, anyways thank you for your help and time!

Summary:
RequestBin POST works.
https://ReqBinID.x.pipedream.net/ttn-eu/api/v2/down/tobipeloratest/forward_gps_data?key=ttn-account-v2.dummy

Changing it to the following doesn’t (“dummy” replaced by correct data of course):
https://integrations.thethingsnetwork.org/ttn-eu/api/v2/down/tobipeloratest/forward_gps_data?key=ttn-account-v2.dummy

[arjanvanb (Arjan)]

@arjanvanb

Can you test without a User-Agent header in Postman?

Some testing shows TTN doesn’t care about a missing User-Agent. The following (along with disabling some headers in Postman’s preferences) creates the same headers as shown in your RequestBin tests, and works just fine in TTN:

Also, it seems that RequestBin shows UTF-8 encoded non-breaking whitespace, which would also increase the Content-Length which does not happen for your tests:

So, to ensure Blynk supports Let’s Encrypt (which I really expect it will…), you might want to test with https://beeceptor.com, which uses Let’s Encrypt just like TTN. (RequestBin uses an Amazon AWS server certificate. Both Beeceptor and RequestBin use a wildcard, while TTN does not.)

[TobiasPe]

@arjanvanb

you might want to test with https://beeceptor.com

Thank you for your help, I really appreciate it. So I did the Test. Postman-Requests are being displayed in Beeceptor but Blynk-Requests sadly aren’t

[TobiasPe]

I‘m running a local Blynk Server. Using the ‚normal‘ Blynk server worked out well.
So, now I will read on how to put Let’s Encrypt on my Rapsberry and post the solution here once I get there.
Any recommendations for readings/tutorials?
Good job, @arjanvanb !

[arjanvanb (Arjan)]

Ah, then I’d also try to find the local server’s logs, which might show clues for future problems?

It seems this server is a Java application? Then you’ll need to make Java trust the Let’s Encrypt Certificate Authority chain. (For this problem, you do not need to make your own instance provide HTTPS, but your instance needs to accept TTN’s server certificate.) Upgrading your Java runtime might already do the trick:

Certificate Compatibility - Let's Encrypt - Free SSL/TLS Certificates

[TobiasPe]

It’s working!!! Updating to the latest build of Java 8 didn’t help (newer versions are not supported by the Blynk server. Eventually also Version 11, but not on the RaspPi).

Had to install and activate Let’s Encrypt Certificates and taking my RaspPi to 2020, time of SSL

Thaaaaaaaaaank you so much @arjanvanb

[TobiasPe]

Another issue in that context: Blynk application needs one of the following answers from TTN HTTP Integration: 200 or 302.
What I get after successful data forwarding: 202.
Is there a way of changing the TTN-Server’s answer to 200?

[arjanvanb (Arjan)]

@TobiasPe

Is there a way of changing the TTN-Server’s answer to 200?

Not without an intermediate server. However, I’d say that is a bug in Blynk, which you might want to report there. Any HTTP 2xx code should really be handled as success, and that should be easy to fix.

[TobiasPe]

@arjanvanb
I addressed the problem to Blynk-Community and one of the founders integrated 202 to the list of accepted HTTP answers just two hours later (very fast ). Now everything is working as expected and I successfully set up data-exchange (through ABP) between Blynk and TTN.
Your help was the biggest contributing factor. Once again thank you!!!