Error when trying to add an End Device from HTTP API

TTN Network and Services The Things Stack v3
1

Hi,
I’m trying to add an End Device from an Android app using the HTTP API.
As indicated in the forum, I have used the console to add a device and I have used Chrome inspector to see the requests sent. I can see the 4 requests for:

  • the Identity Server
  • the Join Server
  • the Network Server
  • the Application Server

I’m now trying to replicate them in my Android App.

Here is what the Console is sending:

{
   "end_device":{
      "ids":{
         "dev_eui":"6895280953AEADD5",
         "join_eui":"ADD5A34886B3E6C8",
         "device_id":"eui6895280953aeadd5"
      },
      "network_server_address":"eu1.cloud.thethings.network",
      "application_server_address":"eu1.cloud.thethings.network",
      "join_server_address":"eu1.cloud.thethings.network"
   },
   "field_mask":{
      "paths":[
         "network_server_address",
         "application_server_address",
         "join_server_address"
      ]
   }
}

Here is what my Android application is sending:

{
   "end_device":{
      "ids":{
         "dev_eui":"6895280953AEADD5",
         "join_eui":"ADD5A34886B3E6C8",
         "device_id":"eui6895280953AEADD5"
      },
      "network_server_address":"eu1.cloud.thethings.network",
      "application_server_address":"eu1.cloud.thethings.network",
      "join_server_address":"eu1.cloud.thethings.network"
   },
   "field_mask":{
      "paths":[
         "network_server_address",
         "application_server_address",
         "join_server_address"
      ]
   }
}

And here is the error that I get:
NetworkUtility.shouldRetryException: Unexpected response code 400 for https://eu1.cloud.thethings.network/api/v3/applications/myloraapplication/devices

Body content:

{
   "code":3,
   "message":"error:pkg/errors:validation (invalid `end_device`: embedded message failed validation)",
   "details":[
      {
         "@type":"type.googleapis.com/ttn.lorawan.v3.ErrorDetails",
         "namespace":"pkg/errors",
         "name":"validation",
         "message_format":"invalid `{field}`: {reason}",
         "attributes":{
            "field":"end_device",
            "name":"CreateEndDeviceRequestValidationError",
            "reason":"embedded message failed validation"
         },
         "correlation_id":"1b3aa9218cd145968ff89251b902105d",
         "cause":{
            "namespace":"pkg/errors",
            "name":"validation",
            "message_format":"invalid `{field}`: {reason}",
            "attributes":{
               "field":"ids",
               "name":"EndDeviceValidationError",
               "reason":"embedded message failed validation"
            },
            "correlation_id":"54422411c0f246ae96005c2d856c6dbd",
            "cause":{
               "namespace":"pkg/errors",
               "name":"validation",
               "message_format":"invalid `{field}`: {reason}",
               "attributes":{
                  "field":"device_id",
                  "name":"EndDeviceIdentifiersValidationError",
                  "reason":"value does not match regex pattern \"^[a-z0-9](?:[-]?[a-z0-9]){2,}$\""
               },
               "correlation_id":"b3243cae98de48dfbcdb67246bd4090b",
               "code":3
            },
            "code":3
         },
         "code":3
      }
   ]
}

Would you have an idea what I could have done wrong?

A side comment:
In the console, I have looked at the 4 requests. The first one is POST and the 3 others are PUT. I was expecting only POST requests.

Thank you!

2

Identifiers are restricted to ^[a-z0-9](?:[-]?[a-z0-9]){2,}$ which means that you cannot use capital letters “AEADD5”.

1 Like
3

Oh, thank you!
I have changed device_id to lower case (“eui6895280953aeadd5”) but for the moment I still have the same error.

Shall I switch also “dev_eui” and “join_eui” to lower case? When I look at what the console is sending, it is uppercase.

What about the underscores like in “network_server_address”?

Thank you

4

All the answers are usually there for the reading …

5

Sorry but I don’t see the problem. Do you mind telling me?

6

Sure!

The response you got back started:

  message:"error:pkg/errors:validation (invalid `end_device`: embedded message failed validation)",

which has the words/phrases error, errors, validation & failed validation in it.

Looking further, you get:

"attributes":{
    "field":"device_id",
    "name":"EndDeviceIdentifiersValidationError",
    "reason":"value does not match regex pattern \"^[a-z0-9](?:[-]?[a-z0-9]){2,}$\""

Which points you at the device_id and that it doesn’t match a regex pattern which tells you the required format for the device_id.

If you don’t read regex, it’s worth learning the basics. And at worse, it could point the way to the docs to read about the device_id format.

The docs and the error messages are very comprehensive and will point the way so you don’t have to break stride to query the inter webs.

7

I have of course tried to read the regex but without much success so I still don’t know.

In the console, “device_id” “eui-6895280953aeadd5” and “eui6895280953aeadd5” are both working. That’s very strange because they don’t work when I use them in my Android application.

I have searched in the documentation some info about “device_id” format (like here Data Formats | The Things Stack for LoRaWAN) but I haven’t found.

As you can see I have searched. Do you mind to tell me what’s wrong ?

Thank you

8

Not sure where this is going but …

Link to the documentation:

https://www.thethingsindustries.com/docs/reference/id-eui-constraints/

Requirements of an ID or EUI

An ID or EUI in The Things Stack must:

  • Have a length of between 3 and 36 characters (inclusive)
    • Exception: User IDs can have a length between 2 and 36 characters (inclusive)
  • Consist of lowercase letters, numbers, and non-consecutive dashes
  • NOT begin or end with a dash

The following regular expression is used to validate IDs and EUIs (with the exception of the User ID):

(^[a-z0-9](?:[-]?[a-z0-9]){2,}$)

As for the regex

(^[a-z0-9](?:[-]?[a-z0-9]){2,}$)

^ = start at the beginning (a very good place to start)

[a-z0-9] = match a to z or 0 - 9 - as this is the first match group, this means the ID has to start with only those characters.

(?:[-]?[a-z0-9]){2,} = match either a dash - or 1 to z or 0 to 9, do that at least 2 times and allow an unlimited number of times.

$ = match end of string - so other characters aren’t included

My hidden agenda

Feed a man a fish and you feed him for a day, teach him to fish and he’s good to code all day long.

Per your previous topic about API’s I believe you’d benefit from one of the secrets of my success: Review the documentation - not to memorise it, but so your neocortex can light up like a Christmas tree when you go back to the docs as your brain won’t surface the exact place and certainly not the exact answer, but it will help eliminate 80% or more of the material so you can concentrate on the most likely locations - like finding your car keys - which always turn out to be in the last place you look.

1 Like
9

Please paste the error messages. It cannot be the same issue if you changed the device_id to satisfy the regex.

10

Oddly enough, I still get the same error. I was suspecting a cache issue so I have added a function call to clear the cache of my requestQueue before doing the new request but it doesn’t help.

Here is the logfile. It shows the request (the “device_id” is “eui-6895280953aeadd5”). Then there is the cache cleared and then there is the error.

2022-02-10 12:58:28.956 1647-1647/com.st.loraprovisioning W/MainActivity: addEndDeviceToIdentityServer: {"end_device":{"ids":{"dev_eui":"6895280953AEADD5","join_eui":"ADD5A34886B3E6C8","device_id":"eui-6895280953aeadd5"},"network_server_address":"eu1.cloud.thethings.network","application_server_address":"eu1.cloud.thethings.network","join_server_address":"eu1.cloud.thethings.network"},"field_mask":{"paths":["network_server_address","application_server_address","join_server_address"]}}
2022-02-10 12:58:28.971 1647-1647/com.st.loraprovisioning D/Volley: [2] DiskBasedCache.clear: Cache cleared.
2022-02-10 12:58:28.985 1647-1647/com.st.loraprovisioning I/Choreographer: Skipped 257 frames!  The application may be doing too much work on its main thread.
2022-02-10 12:58:29.159 1647-1703/com.st.loraprovisioning E/Volley: [52] NetworkUtility.shouldRetryException: Unexpected response code 400 for https://eu1.cloud.thethings.network/api/v3/applications/myloraapplication/devices
2022-02-10 12:58:29.166 1647-1647/com.st.loraprovisioning E/MainActivity: onErrorResponse: null
2022-02-10 12:58:39.741 1647-1647/com.st.loraprovisioning E/MainActivity: body: {"code":3,"message":"error:pkg/errors:validation (invalid `end_device`: embedded message failed validation)","details":[{"@type":"type.googleapis.com/ttn.lorawan.v3.ErrorDetails","namespace":"pkg/errors","name":"validation","message_format":"invalid `{field}`: {reason}","attributes":{"field":"end_device","name":"CreateEndDeviceRequestValidationError","reason":"embedded message failed validation"},"correlation_id":"a09ff68deabd4d50ab03f77bfa0945ef","cause":{"namespace":"pkg/errors","name":"validation","message_format":"invalid `{field}`: {reason}","attributes":{"field":"ids","name":"EndDeviceValidationError","reason":"embedded message failed validation"},"correlation_id":"d2c3df6893ae447784c3dc837c7070f6","cause":{"namespace":"pkg/errors","name":"validation","message_format":"invalid `{field}`: {reason}","attributes":{"field":"device_id","name":"EndDeviceIdentifiersValidationError","reason":"value does not match regex pattern \"^[a-z0-9](?:[-]?[a-z0-9]){2,}$\""},"correlation_id":"9c67737287ac4367b974e9def203abba","code":3},"code":3},"code":3}]}

Here is the response body after formatting:

{
   "code":3,
   "message":"error:pkg/errors:validation (invalid `end_device`: embedded message failed validation)",
   "details":[
      {
         "@type":"type.googleapis.com/ttn.lorawan.v3.ErrorDetails",
         "namespace":"pkg/errors",
         "name":"validation",
         "message_format":"invalid `{field}`: {reason}",
         "attributes":{
            "field":"end_device",
            "name":"CreateEndDeviceRequestValidationError",
            "reason":"embedded message failed validation"
         },
         "correlation_id":"a09ff68deabd4d50ab03f77bfa0945ef",
         "cause":{
            "namespace":"pkg/errors",
            "name":"validation",
            "message_format":"invalid `{field}`: {reason}",
            "attributes":{
               "field":"ids",
               "name":"EndDeviceValidationError",
               "reason":"embedded message failed validation"
            },
            "correlation_id":"d2c3df6893ae447784c3dc837c7070f6",
            "cause":{
               "namespace":"pkg/errors",
               "name":"validation",
               "message_format":"invalid `{field}`: {reason}",
               "attributes":{
                  "field":"device_id",
                  "name":"EndDeviceIdentifiersValidationError",
                  "reason":"value does not match regex pattern \"^[a-z0-9](?:[-]?[a-z0-9]){2,}$\""
               },
               "correlation_id":"9c67737287ac4367b974e9def203abba",
               "code":3
            },
            "code":3
         },
         "code":3
      }
   ]
}
11

This is for sure a cache issue. I have provided an empty json object to the request and I still get the same error. I’m searching why…

12

Try creating it in the console to see if there is a validation error there.

And as you’ve already spotted, you can see the to and fro in the console log of your browser.

13

Plus, here’s a list of all the threads I can quickly find - many of which ended with a positive outcome, a few using my Python & PHP - but I haven’t got to doing it via JS yet.

https://www.thethingsnetwork.org/forum/search?q=join_server_address

Another thing to try is a totally different device_id format to see what happens - it may throw some light on the situation.

14

Does it work on a desktop ie, not a mobile device?

Most mobile providers do a lot of proxying, caching, firewalling and general prevention of data flow unless it suits them. So it wouldn’t surprise me if there was some hidden proxy/cache in the mix. You could try mixing up the URL by appending different path parameters. Or try a WiFi only connection.

Almost all my stuff talks to my servers which can then be sure of executing a sequence of commands - with mobile there is a real risk that you get one or two transactions done and then the network glitches. I can’t remember where, but using the JS API isn’t recommended if you can do it any other way - like via the CLI.

15

While playing with curl, I have seen that if I post the same request but with an empty JSON object ("{}"), I get the same response complaining about device_id value “not matching regex pattern”:

{
    "code": 3,
    "message": "error:pkg/errors:validation (invalid `end_device`: embedded message failed validation)",
    "details": [{
        "@type": "type.googleapis.com/ttn.lorawan.v3.ErrorDetails",
        "namespace": "pkg/errors",
        "name": "validation",
        "message_format": "invalid `{field}`: {reason}",
        "attributes": {
            "field": "end_device",
            "name": "CreateEndDeviceRequestValidationError",
            "reason": "embedded message failed validation"
        },
        "correlation_id": "553ef3bb85e841fbafb7fa1191a7c0b9",
        "cause": {
            "namespace": "pkg/errors",
            "name": "validation",
            "message_format": "invalid `{field}`: {reason}",
            "attributes": {
                "field": "ids",
                "name": "EndDeviceValidationError",
                "reason": "embedded message failed validation"
            },
            "correlation_id": "ab3856cd3ea042fb800a1f8dca5dbc4a",
            "cause": {
                "namespace": "pkg/errors",
                "name": "validation",
                "message_format": "invalid `{field}`: {reason}",
                "attributes": {
                    "field": "device_id",
                    "name": "EndDeviceIdentifiersValidationError",
                    "reason": "value does not match regex pattern \"^[a-z0-9](?:[-]?[a-z0-9]){2,}$\""
                },
                "correlation_id": "fac0fff4af1d49959696d8ac24a3e47d",
                "code": 3
            },
            "code": 3
        },
        "code": 3
    }]
}

This is a bit missleading. Would it be possible to change the server so that it complains about the absence of “device_id” field rather than complaining about a wrong value, (which is not correct)?

16

Would it be possible to change the server so that it complains about the absence of “device_id”

Our validation layer only checks for missing structs/repeated fields and not on strings/bytes. That’s what our validator supports and we won’t change that anytime soon.

rather than complaining about a wrong value, (which is not correct)?

I don’t see the issue here. If the error message points you towards the “device_id” field, then you’re going to check that field whether it’s incorrect or empty. BTW, an empty field does not satisfy the regex so this is not incorrect.

17

OK, it’s like laundry day here, so if you give me the JSON with placeholders for the EUI’s and key and device_id I can take it for a spin - I’d need a JS code base at some point so may as well crack it out now.

18

Thanks. NB: I’m using Java and not JS.

Here is my code:

    private void addEndDeviceToIdentityServer() {

        JSONObject dataObject = new JSONObject();

        // Prepare JSON data
        try {
            JSONObject idsObject = new JSONObject();
            idsObject.put("dev_eui", mDevEUI);
            idsObject.put("join_eui", mJoinEUI);
            idsObject.put("device_id", mDeviceId);

            JSONObject endDeviceObject = new JSONObject();
            endDeviceObject.put("ids", idsObject);
            endDeviceObject.put("network_server_address", TTN_SERVER);
            endDeviceObject.put("application_server_address", TTN_SERVER);
            endDeviceObject.put("join_server_address", TTN_SERVER);

            JSONArray paths = new JSONArray();
            paths.put("network_server_address");
            paths.put("application_server_address");
            paths.put("join_server_address");

            JSONObject fieldMaskObject = new JSONObject();
            fieldMaskObject.put("paths", paths);

            dataObject.put("end_device", endDeviceObject);
            dataObject.put("field_mask", fieldMaskObject);

        } catch (JSONException e) {
            e.printStackTrace();
            return;
        }

        Log.w(TAG,"addEndDeviceToIdentityServer: " + dataObject);

        mRequestQueue = Volley.newRequestQueue(this);
        mRequestQueue.getCache().clear();
        
        String url = TTN_URL + "/applications/" + mApplicationId + "/devices";

        JsonObjectRequest jsonObjReq = new JsonObjectRequest(Request.Method.POST, url, dataObject,
                new Response.Listener<JSONObject>() {

                    @Override
                    public void onResponse(JSONObject response) {
                        Log.d(TAG, response.toString());
                    }
                }, new Response.ErrorListener() {

            @Override
            public void onErrorResponse(VolleyError error) {
                if (error == null || error.networkResponse == null) {
                    return;
                }
                Log.e(TAG, "onErrorResponse: " + error.getMessage());

                String body;
                //get status code here
                final String statusCode = String.valueOf(error.networkResponse.statusCode);
                //get response body and parse with appropriate encoding
                try {
                    body = new String(error.networkResponse.data,"UTF-8");
                    Log.e(TAG, "body: " + body);
                } catch (UnsupportedEncodingException e) {
                    e.printStackTrace();
                }
            }
        }) {
            @Override
            public Map<String, String> getHeaders() throws AuthFailureError {
                final Map<String, String> headers = new HashMap<>();
                headers.put("Centent-Type", "application/json");
                headers.put("Authorization", "Bearer " + mMyAppApiKey);
                return headers;
            }
        };

        jsonObjReq.setShouldCache(false);
        jsonObjReq.setTag(TAG);
        
        // Adding request to request queue
        mRequestQueue.add(jsonObjReq);

    }

And here is the JSON object:
{"end_device":{"ids":{"dev_eui":"6895280953AEADD5","join_eui":"ADD5A34886B3E6C8","device_id":"eui-6895280953aeadd5"},"network_server_address":"eu1.cloud.thethings.network","application_server_address":"eu1.cloud.thethings.network","join_server_address":"eu1.cloud.thethings.network"},"field_mask":{"paths":["network_server_address","application_server_address","join_server_address"]}}

19

Opps! OK, I’m still game, because if I can’t hack together some Java based on my fleeting use of it 20 years ago, I’ll hang up my programming trousers. Probably be this evening whilst watching a random submarine movie.

20

I have fixed the issue, I will post an explanation in a while.
So you will be able to enjoy your movie without disturbance Nick ;o)