Mqtt - node red - grafana - lorasoundsensor

I know this is probably not the right forum to adress my ignorance.

For the past 6 months I have been working on developing a noise monitoring network such as described:

I purchased and configured a TTN gateway for this purpose.

I am currently at the point where I can pull in data in node red via mqtt and save it as json.

I would like to get the script to convert it into a CSV file via Node Red and to import it into Mariadb, I don’t know enough about C+ to spend a lot of time for another trial on try and error hours on this.

Knowing that I only got the gateway connected after 3 months, this says enough :slight_smile:

The internet has become such a maze that I waste more time looking up and trying things out. More errors and trying than usable results … .

If Marcel Meeks, my source and inspiration, could share this code here or via GitHub, I would be very grateful ! ! !

Tips on soundrecognition are also helpful (sound of industries, buldozer, betoncrusher).

That’s an X-Y problem - what you actually want to do is to turn the JSON in to something that can be inserted in to MariaDB - the intermediary step of thinking in terms of CSV will be clouding the issue on your searches, which, as you say, will be cluttered with Grap (Google Cr_p).

As MariaDB is a relatively new thing compared to all the mySQL blog posts, so I’d also think about using that as a keyword rather than MariaDB.

And search this forum as it does have some hits as, well, this is where most people doing this sort of stuff are likely to be. This older post may help: HTTP integration - #26 by Charles, the participants from that point of that topic do know stuff.

If you are willing to share your noise data, I would consider to share it with sensor.community. The dutch public health institution RIVM will then automatically pick it up from there and display it on their samenmeten map too, Samen Meten - Dataportaal
The sensor.community people will also allow you to read back your own data from their archives (although I don’t know all the details about this myself).

I wrote a forwarder in Java that subscribes to the TTN MQTT uplink topic, decodes the sensor information and forwards it so sensor.community:

This forwarder has been tested to work with the data format from Marcel Meek’s sensor (you do need to configure a payload decoder in the TTN application/device).

I have an instance of this forwarder running already, I can add your sensors too if you want. For this I require the name of the application and a TTN API key with rights to receive MQTT messages from your devices and to read device properties. You can also run this forwarder by yourself too, the docker/ subdirectory contains a docker-compose file.

You need to define a noise sensor at https://devices.sensor.community/ with the right properties (noise sensor) and obtain a unique device id that can then be linked to your device by setting it as a TTN device attribute ‘senscom-id’.

A noise sensor on LoRaWAN/TTN has some challenges:
The sensor.community map runs on a 5 minute heart-beat. You’ll probably not be able to guarantee a 5 minute interval because of fair-use constraints of TTN. It might just fit with a small payload and communication at SF7 (nearby gateway). So your sensor may not always be visible on their map (although that might not be a problem for you).

i will use the following building blocks

noise sensor lorawan => ttn => mqtt => node-red => influxdb => grafana

you seem to have the first 4 steps covered

where are your node-red hosted

since your noise data are time series based i will rather use a timeseries database - influxdb

and then finally where are you hosting your grafana

and for us to help you we need to know what your data looks like, the json

Thanks Piet for willing to help, much appreciated ! ! !

I will try to answer your questions below:
8 Dec 16:38:39 - [info] [mqtt-broker:9c448ad…] Connected to broker: schollebeek@mqtt://eu1.cloud.thethings.industries:1883

I have to study influxdb (again lots of try and error ).
The reason for using mariadb or sql is that i want to backup all the readings from the 6 sensors.

------------------json file {
“end_device_ids”: {
“device_id”: “eui-000044959edad4d4”,
“application_ids”: {
“application_id”: “soundsensor”
},
“dev_eui”: “000044959EDAD4D4”,
“join_eui”: “000044959EDAD4D4”,
“dev_addr”: “2608EA2A”
},
“correlation_ids”: [
“gs:uplink:01HH5354SBKJ…H60”
],
“received_at”: “2023-12-08T16:11:17.883569641Z”,
“uplink_message”: {
“session_key_id”: “AYwhyRj3DFMFV…”,
“f_port”: 22,
“f_cnt”: 11449,
“frm_payload”: “OZDSrrL+z7T/0IWhwbidk397fg==”,
“decoded_payload”: {
“la”: {
“avg”: 38.89411764705883,
“max”: 46.94117647058824,
“min”: 32.188235294117646,
“spectrum”: [
-9.670588235294115,
9.788235294117651,
27.041176470588233,
32.529411764705884,
31.894117647058824,

Grafana:

You are subscribed to Grafana Cloud Free. Your subscription includes:

  • Grafana Instance
  • Prometheus Endpoint
  • Graphite Endpoint
  • Logs Backend
  • Traces Backend
  • Performance Testing
  • Unlimited Dashboards
  • Community Support

Grafana —> if you have further questions shoot!

Thanks for the support!

Thanks Bertrik for willing to help ! Much appreciated !

I will look into SAMEN METEN.
I allready visited their website.
Important for me is that i can keep my readings (back-up).
The wabsite concerning the flight traffic and noise fron Schiphol has my intrests to i am not sure that i can combine samen meten sith soundsensor and flighttraffic or detecting industriesounds like a concretecrusher … .

I will look into your sensor data bridge to.

One step at a time …
Thanks for the help provided!

you need the use </> tool to post code/text/json all the text base stuff

influxdb are fairly straight forward 1.8 is a bit easier to use than 2.x if you come from a mysql background

it is neither here or there

any db does not matter the flavor you can backup 6 or 6000 sensors makes no difference

Next try :slight_smile:

{
“end_device_ids”: {
“device_id”: “eui-0000d8929edad4d4”,
“application_ids”: {
“application_id”: “soundsensor”
},
“dev_eui”: “0000D8929EDAD4D4”,
“join_eui”: “0000D8929EDAD4D4”,
“dev_addr”: “26080D76”
},
“correlation_ids”: [
“gs:uplink:01HH59XTMBEZJK09ET8MEMKCGF”
],
“received_at”: “2023-12-08T18:09:38.140186091Z”,
“uplink_message”: {
“session_key_id”: “AYwwqMZv/GQQzB5z+L61lw==”,
“f_port”: 22,
“f_cnt”: 7415,
“frm_payload”: “MbnWx9L849T/5aW7zMCft5yAiw==”,
“decoded_payload”: {
“la”: {
“avg”: 38.23921568627451,
“max”: 41.12156862745098,
“min”: 35.54901960784314,
“spectrum”: [
-7.694117647058821,
9.733333333333338,
23.1,
28.294117647058826,
27.35294117647059,
35.164705…

In the box that you type the messages, there is a toolbar with B and I and so on. In the middle there is a tool with </> - that’s the tool to use when you have selected the text you have pasted - which is currently being truncated for some reason - we’ll need all of it, it doesn’t stop at 35.164705… there will be a heap more.

this is how easy it is to use node-red to write date to influxdb.

the inject node simulates the mqtt in and the debug node is where the influxdb node will be

in the inject node you will see the complex json that a ttn sends to you

[
    {
        "id": "e95f216c4a3174af",
        "type": "inject",
        "z": "55354f631a16938a",
        "name": "this will be the mqtt in node",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "{\"end_device_ids\":{\"device_id\":\"eui-thisIsTheEUI\",\"application_ids\":{\"application_id\":\"myApp\"},\"dev_eui\":\"eui-thisIsTheEUI\",\"join_eui\":\"eui-thisIsTheJoinEUI\",\"dev_addr\":\"260B090E\"},\"correlation_ids\":[\"gs:uplink:xxxxxxxxx\"],\"received_at\":\"2023-12-08T20:20:55.378199900Z\",\"uplink_message\":{\"session_key_id\":\"xxxxxxxxxx\",\"f_port\":85,\"f_cnt\":42932,\"frm_payload\":\"xxxxx\",\"decoded_payload\":{\"door\":0,\"humidity\":86,\"temperature\":20.4},\"rx_metadata\":[{\"gateway_ids\":{\"gateway_id\":\"oneCloseToYou\",\"eui\":\"gatewayEUI\"},\"timestamp\":2686651371,\"rssi\":-92,\"channel_rssi\":-92,\"snr\":8.2,\"location\":{\"latitude\":1.10101,\"longitude\":2.20202,\"altitude\":615000,\"source\":\"SOURCE_REGISTRY\"},\"uplink_token\":\"xxxxxxxxxxx\",\"channel_index\":4,\"received_at\":\"2023-12-08T20:20:55.070893669Z\"},{\"gateway_ids\":{\"gateway_id\":\"packetbroker\"},\"packet_broker\":{\"message_id\":\"xxxxxxx\",\"forwarder_net_id\":\"000013\",\"forwarder_tenant_id\":\"xxxxxx\",\"forwarder_cluster_id\":\"eu1.cloud.thethings.industries\",\"forwarder_gateway_eui\":\"xxxxxxx\",\"forwarder_gateway_id\":\"xxxxxxxx\",\"home_network_net_id\":\"000013\",\"home_network_tenant_id\":\"ttn\",\"home_network_cluster_id\":\"eu1.cloud.thethings.network\"},\"time\":\"2023-12-08T20:20:55.061923Z\",\"rssi\":-115,\"channel_rssi\":-115,\"snr\":-2.25,\"location\":{\"latitude\":2.20202,\"longitude\":1.10101,\"altitude\":95000},\"uplink_token\":\"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\",\"received_at\":\"2023-12-08T20:20:55.168940077Z\"}],\"settings\":{\"data_rate\":{\"lora\":{\"bandwidth\":125000,\"spreading_factor\":7,\"coding_rate\":\"4/5\"}},\"frequency\":\"867300000\",\"timestamp\":2686651371},\"received_at\":\"2023-12-08T20:20:55.169445657Z\",\"consumed_airtime\":\"0.061696s\",\"version_ids\":{\"brand_id\":\"xxxxx\",\"model_id\":\"xxxxxx\",\"hardware_version\":\"V3.0\",\"firmware_version\":\"1.2\",\"band_id\":\"EU_863_870\"},\"network_ids\":{\"net_id\":\"000013\",\"ns_id\":\"xxxxxxx\",\"tenant_id\":\"ttn\",\"cluster_id\":\"eu1\",\"cluster_address\":\"eu1.cloud.thethings.network\"}}}",
        "payloadType": "json",
        "x": 290,
        "y": 1040,
        "wires": [
            [
                "0f7519882a50eb93"
            ]
        ]
    },
    {
        "id": "700eb8752c1c36cb",
        "type": "debug",
        "z": "55354f631a16938a",
        "name": "debug - this will be to influxdb",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "x": 910,
        "y": 1040,
        "wires": []
    },
    {
        "id": "0f7519882a50eb93",
        "type": "change",
        "z": "55354f631a16938a",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "payload.dev_eui",
                "pt": "msg",
                "to": "payload.end_device_ids.dev_eui",
                "tot": "msg"
            },
            {
                "t": "set",
                "p": "payload.battery",
                "pt": "msg",
                "to": "payload.uplink_message.decoded_payload.battery",
                "tot": "msg"
            },
            {
                "t": "set",
                "p": "payload.humidity",
                "pt": "msg",
                "to": "payload.uplink_message.decoded_payload.humidity",
                "tot": "msg"
            },
            {
                "t": "set",
                "p": "payload.temperature",
                "pt": "msg",
                "to": "payload.uplink_message.decoded_payload.temperature",
                "tot": "msg"
            },
            {
                "t": "delete",
                "p": "payload.end_device_ids",
                "pt": "msg"
            },
            {
                "t": "delete",
                "p": "payload.correlation_ids",
                "pt": "msg"
            },
            {
                "t": "delete",
                "p": "payload.uplink_message",
                "pt": "msg"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 500,
        "y": 1040,
        "wires": [
            [
                "ae3ff2a21e492806"
            ]
        ]
    },
    {
        "id": "ae3ff2a21e492806",
        "type": "function",
        "z": "55354f631a16938a",
        "name": "dew point",
        "func": "let h = msg.payload.humidity\nlet t = msg.payload.temperature\n\nlet dew = t - ((100 - h)/5)\n\nmsg.payload.dewPoint = dew\n\n\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 680,
        "y": 1040,
        "wires": [
            [
                "700eb8752c1c36cb"
            ]
        ]
    }
]

Gelukt !
I had to put it in a file en then i could copy it …

The only fields i need in influxdb are device-id, timestamp and all of the dB-values to make it visual in grafana.


{
    "end_device_ids": {
        "device_id": "eui-000044959edad4d4",
        "application_ids": {
            "application_id": "soundsensor"
        },
        "dev_eui": "000044959EDAD4D4",
        "join_eui": "000044959EDAD4D4",
        "dev_addr": "2608EA2A"
    },
    "correlation_ids": [
        "gs:uplink:01HH5M9HD670KTK9QC557"
    ],
    "received_at": "2023-12-08T21:10:47.668690592Z",
    "uplink_message": {
        "session_key_id": "AYwhyRj3DF",
        "f_port": 22,
        "f_cnt": 11752,
        "frm_payload": "Rnz7qqX/w6f/xZ+ztJqYipaOcg==",
        "decoded_payload": {
            "la": {
                "avg": 46.66666666666667,
                "max": 68.90196078431373,
                "min": 34.03921568627451,
                "spectrum": [
                    4.247058823529414,
                    22.937254901960788,
                    33.311764705882354,
                    33.67450980392157,
                    38.52549019607843,
                    37.88235294117647,
                    42.3764705882353,
                    39.98039215686275,
                    30.194117647058825
                ]
            },
            "lc": {
                "avg": 53.529411764705884,
                "max": 70,
                "min": 45.294117647058826,
                "spectrum": [
                    40.64705882352941,
                    48.33725490196079,
                    49.21176470588235,
                    42.274509803921575,
                    41.72549019607843,
                    37.88235294117647,
                    41.3764705882353,
                    39.280392156862746,
                    28.294117647058826
                ]
            },
            "lz": {
                "avg": 54.078431372549026,
                "max": 70,
                "min": 45.84313725490196,
                "spectrum": [
                    43.64705882352941,
                    49.13725490196079,
                    49.411764705882355,
                    42.274509803921575,
                    41.72549019607843,
                    37.88235294117647,
                    41.1764705882353,
                    38.98039215686275,
                    31.294117647058826
                ]
            }
        },
        "rx_metadata": [
            {
                "gateway_ids": {
                    "gateway_id": "eui-9c53a3f6065ecb2f",
                    "eui": "9C53A3F6065ECB2F"
                },
                "timestamp": 3854196017,
                "rssi": -73,
                "channel_rssi": -73,
                "snr": 10.5,
                "frequency_offset": "2777",
                "uplink_token": "CiIKIAoUZXVpLTljNTNhM2Y2MDY1ZWNiMmYSCJx6",
                "channel_index": 1,
                "received_at": "2023-12-08T21:10:47.445691610Z"
            }
        ],
        "settings": {
            "data_rate": {
                "lora": {
                    "bandwidth": 125000,
                    "spreading_factor": 7,
                    "coding_rate": "4/5"
                }
            },
            "frequency": "868300000",
            "timestamp": 3854196017
        },
        "received_at": "2023-12-08T21:10:47.463056080Z",
        "consumed_airtime": "0.071936s",
        "network_ids": {
            "net_id": "000013",
            "ns_id": "EC656E0000000101",
            "tenant_id": "schollebeek",
            "cluster_id": "eu1",
            "cluster_address": "eu1.cloud.thethings.industries",
            "tenant_address": "schollebeek.eu1.cloud.thethings.industries"
        }
    }
}
{
    "end_device_ids": {
        "device_id": "eui-00002c2c8c08b764",
        "application_ids": {
            "application_id": "soundsensor"
        },
        "dev_eui": "00002C2C8C08B764",
        "join_eui": "00002C2C8C08B764",
        "dev_addr": "260883D8"
    },
    "correlation_ids": [
        "gs:uplink:01HH5MAE86PE5GRMS3"
    ],
    "received_at": "2023-12-08T21:11:17.206592611Z",
    "uplink_message": {
        "session_key_id": "AYw0zz+pLblmMW",
        "f_port": 22,
        "f_cnt": 6374,
        "frm_payload": "P7baxsj63cr/37zNv7iuvaN0bg==",
        "decoded_payload": {
            "la": {
                "avg": 48.91764705882353,
                "max": 53.858823529411765,
                "min": 44.964705882352945,
                "spectrum": [
                    7.047058823529419,
                    24.447058823529414,
                    31.088235294117645,
                    36.858823529411765,
                    39.78823529411765,
                    46.694117647058825,
                    41.47058823529412,
                    29.658823529411766,
                    26.076470588235296
                ]
            },
            "lc": {
                "avg": 54.6,
                "max": 61.76470588235294,
                "min": 49.411764705882355,
                "spectrum": [
                    43.44705882352942,
                    49.847058823529416,
                    46.98823529411764,
                    45.45882352941177,
                    42.98823529411765,
                    46.694117647058825,
                    40.47058823529412,
                    28.958823529411767,
                    24.176470588235297
                ]
            },
            "lz": {
                "avg": 55.09411764705882,
                "max": 63,
                "min": 49.90588235294118,
                "spectrum": [
                    46.44705882352942,
                    50.64705882352941,
                    47.188235294117646,
                    45.45882352941177,
                    42.98823529411765,
                    46.694117647058825,
                    40.27058823529412,
                    28.658823529411766,
                    27.176470588235297
                ]
            }
        },
        "rx_metadata": [
            {
                "gateway_ids": {
                    "gateway_id": "eui-9c53a3f6065ecb2f",
                    "eui": "9C53A3F6065ECB2F"
                },
                "timestamp": 3883738917,
                "rssi": -50,
                "channel_rssi": -50,
                "snr": 13.8,
                "frequency_offset": "-2551",
                "uplink_token": "CiIKIAoUZXVpLTljNTNhM2Y2M
            }
        ],
        "settings": {
            "data_rate": {
                "lora": {
                    "bandwidth": 125000,
                    "spreading_factor": 7,
                    "coding_rate": "4/5"
                }
            },
            "frequency": "868100000",
            "timestamp": 3883738917
        },
        "received_at": "2023-12-08T21:11:16.999454827Z",
        "consumed_airtime": "0.071936s",
        "locations": {
            "user": {
                "latitude": 51.120161367612916,
                "longitude": 4.584496288331605,
                "source": "SOURCE_REGISTRY"
            }
        },
        "network_ids": {
            "net_id": "000013",
            "ns_id": "EC656E0000000101",
            "tenant_id": "schollebeek",
            "cluster_id": "eu1",
            "cluster_address": "eu1.cloud.thethings.industries",
            "tenant_address": "schollebeek.eu1.cloud.thethings.industries"
        }
    }
}

Bertrik,

I asume that MQTT sends its data to al the listeners (at acertain ip-adres) that are registerd to listen to the specified sensor.

I am not in to docker yet so it would be nice if you could add my sensor (just the one in front of my home) to the community.
“dev_eui”: “00002C2C8C08B764”,
“join_eui”: “00002C2C8C08B764”,
“dev_addr”: “26087D67”

eu1.cloud.thethings.industries:8883
soundsensor@schollebeek
Right below the password for MQTT
<redacted, sort of>

A new API key has just been created for your application soundsensor on Schollebeek.

API Key ID: <redacted, sort of>

API Key Name: <redacted, sort of>
Rights:

RIGHT_APPLICATION_TRAFFIC_READ
read application traffic (uplink and downlink)
RIGHT_APPLICATION_TRAFFIC_DOWN_WRITE
write downlink application traffic

Do you need anything else?
Thank you for helping me out !

A very urgent course in cyber security - you only needed to provide a read application traffic key but you’ve provided a key, in public, that allows people to try to downlink to your device - pity the device being overwhelmed with a downlink for every up line and the gateway(s) doing so. A denial of service for TTN in a local area.

So please revoke (aka delete) that API key and send the details via private message.

Descartes,

it occurd to me that it was risky - i think Bertrik took the neccesary steps already - thanks by the way !

Thank you for making it very clear !

I very much doubt he was able to revoke a key that is in your account.

If you have not revoked the key you put in the post, you are compromising your account and could impact the TTN community servers.

Please confirm you have revoked the key.

I’d note for those reading along, that Google is very pro-active in indexing this site, so anything said here doesn’t stay here, it can exist on the internet until the end of time.

Descartes,
the key are no longer available in this post.
I am sorry for the discomfort.

Actually, yes they are! Edits can be viewed and ‘replayed’ so simply deleting/redacting in you message in the thread is not achieving what Nick tried to caution you over and what he advised. You need to rescind the keys at source such that they are no longer valid and can never be used again :wink:

NNSXS6IVG7VQC3UF22H32LNZ5WIC4XFZFS####…###BCSHSQ

A new API key has just been created for your application soundsensor on Schollebeek.

API Key ID: 6IVG7VQC3UF#####......####GCKXA

API Key Name: mqtt-password-key-170###.....###437

Just to demonstrate; ###…#### to mask of course :slight_smile:
You would be wise to follow the advice Nicks gives…