V3 console Live Data filtering

jezd · February 6, 2021, 3:13pm

Web browser scroll bars in console are a nightmare in V3 compared to V2 console, must be set at 25% of those is V2 (ie normal)

mathieu_pouillot · May 18, 2021, 10:36am

+1 to the filter option. It is a real lack on this release to debug quickly just to show simple uplink, downlink, join,…

best regards,

Mathieu

descartes · May 18, 2021, 10:39am

I’m working on live data tool to help with this - code named FireHose (as in, try drinking from one).

I’ll be needing some testers shortly, however they will have to be able read instructions and deal with a grumpy sarcastic old developer.

Jeff-UK · May 18, 2021, 10:44am

Unlike the current tester

descartes · May 18, 2021, 10:45am

But he does cope well with the grumpy sarcastic old developer!

Birds of a feather?

kschiffer · May 18, 2021, 11:34am

Hey everyone!

Note that filtering out verbose events by default is a feature coming up in the next release which will be deployed shortly thereafter. It is a first step towards improving the usability of the event views in the Console (as in taming the firehose ).

Filter out verbose events in the Console

TheThingsNetwork:v3.13 ← TheThingsNetwork:feature/2231-event-verbosity-filter

opened 05:14PM - 10 May 21 UTC

kschiffer

+277 -21

#### Summary This PR will apply a toggle-able verbosity filter to the event s…tream which will by default filter out all events that are not typically of direct interest. ![](https://user-images.githubusercontent.com/5710611/117699085-b4cd6c80-b1c4-11eb-8bbf-5abb8c010d8c.png) References #2231 With verbose mode deactivated (default) only the following events will be ingested: ```javascript export const EVENT_VERBOSE_FILTERS = [ 'as.*.drop', 'as.down.data.forward', 'as.up.data.forward', 'gs.down.send', 'gs.gateway.connect', 'gs.gateway.disconnect', 'gs.status.receive', 'gs.up.receive', 'js.join.accept', 'js.join.reject', 'ns.mac.*.answer.reject', '*.warning', '*.fail', 'organization.*', 'user.*', 'gateway.*', 'application.*', 'end_device.*', 'client.*', 'oauth.*', ] ``` #### Changes * Add a constant module for event filters * Update store logic for applying filters to event streams * Update the `<Events />` component to enable filtering out verbose events * Update entity event containers accordingly #### Testing Manual testing in the staging environment ##### Regressions This could affect event ingestion in the Console #### Notes for Reviewers * I did not add the filter for organization events, since the filter would have no effect. For completeness I added the redux primitives for organizations though. * I have implemented this filter at the ingestion level within the reducer, meaning that filtered events will not be added to the store. This means that historical verbose events can not be toggled, since they were already discarded. I've decided to go this route to also lift some weight on the whole event store logic (see also #2887). With this implementation it will be possible to examine events for entities with a higher event frequency. * Later on, other filter could be applied on top of this but filtered out on the component level so that the visibility can be toggled for past events as well. #### Checklist * [x] Scope: The referenced issue is addressed, there are no unrelated changes. * [x] Compatibility: The changes are backwards compatible with existing API, storage, configuration and CLI, according to the compatibility commitments in `README.md` for the chosen target branch. * [x] Documentation: Relevant documentation is added or updated. * [x] Changelog: Significant features, behavior changes, deprecations and fixes are added to `CHANGELOG.md`. * [x] Commits: Commit messages follow guidelines in `CONTRIBUTING.md`, there are no fixup commits left.

descartes · May 18, 2021, 11:52am

Cool - your’s is Lego Technic - mine is very much Lego Duplo.

@kschiffer, assuming you are at the heart of the Matrix, can you explain the correlation ids please - what are they, how might we use them and more importantly, particularly for mobile apps, can we turn them off?

kschiffer · May 18, 2021, 12:03pm

Cool - your’s is Lego Technic - mine is very much Lego Duplo.

Not sure if that is good or bad. You mean that you would like to have even less data displayed (even more data filtered out)?

Correlation IDs are, as the name suggests, IDs that allow you to link an event to other event based on a specific topic. E.g. for gateways, you’ll see a correlation ID for the connection, allowing you to identify all gateway events that are related to this particular gateway connection. Another topic would be e.g. an uplink. You will see that all gateway events relating to the same uplink will share the same correlation ID starting with gs:uplink:.

In the future we will use these IDs in the Console to highlight related events.

I’m not exactly sure what you mean by “turning them off”. If you mean to make the event stream omit the correlation IDs (or any other event property), then yes, something like this is in consideration.

descartes · May 18, 2021, 12:24pm

Both - I like yours for the detail but when I’m running round the garden pouring water on to sensors, I’d rather have a really simple display that just shows me that an uplink was triggered which means the alarm was triggered. I’ve got something that replicates the v2 traffic view without the expanding detail working, just needs a bit more tidy up before beta-testing.

Plus mobile needs the bare minimum unless you have an A2 side iPad Air.

But when it gets gritty, return to “The Source” - the official console - what would be considered “The Truth”.

I’d not worry about the many variants of live data that people could aspire to - the API’s are there for people to roll their own and it means the wife is happy because I’m fully occupied.

Understand the correlation id’s now - very good - but mostly they are wasted bytes for the average console view - if I run out of things to do I could get some API feeds running and have different screens linking the entries via the correlation-ids - probably need to be a glass freestanding monitor with gesture control so I can swipe widgets around.

kschiffer · May 18, 2021, 12:33pm

I see. Yeah indeed you’re always free to come up with your own interface that suits your specific needs (which is part of the power of TTS). Obviously though in the Console we try to come up with a design that suits as many needs as possible. As is known, we’re not quite there yet, but improvements will come in steadily now.

Btw, regarding mobile, the TTS Console is fully responsive and also the event views are optimized for mobile screensizes (obviously with some limitations). We will flesh this out further with the ability to hide columns as well, but it can already be used pretty smoothly (imho).

descartes · May 18, 2021, 12:43pm

Please don’t try - @kersing or @Jeff-UK tell him - it’s not possible.

I think all we need short term is a switch to only show join, uplink & downlinks.

Yes, aware it’s all responsive - but again, use cases vary - try horizontal sleet with wind chill at -15℃ - fingers too cold to swipe etc etc.

kschiffer · May 18, 2021, 12:49pm

I’m very much aware of the trade-off. Still what I mean is a sane approximation that serves the most common use cases without sacrificing usability. Any choices there need to be made with careful consideration. We’re not planing to bloat event streams with millions of options just to cover every last use case.

This is coming up.

That is the kind of “every last use case” use case I was thinking of .

Jeff-UK · May 18, 2021, 7:58pm

Ok guys all sounds good but in last case please dont ask/expect me to be the tester (though British summer will likely be fitting environmental test! )

descartes · May 18, 2021, 8:24pm

Standard Operational Procedure for flying gliders in the Peak District in winter - we log flights on computer.

Once you get up in the air, the flying greenhouse / canopy keeps you warm along with electric socks.

ebenl · September 14, 2022, 10:50pm

Hello, I’m sending data to my GW, I can see the activity counter change in TTN, but the LIVE DATA never shows anything?

descartes · September 15, 2022, 7:52am

As per many other posts asking similar, this is not updated in absolute real time due to the constraints / load on the servers …

ebenl · September 15, 2022, 8:29am

Thanks Nick

The thing is that I never see any “live data”. I understand that it can’t be realtime, just strange that nothing ever shows.

descartes · September 15, 2022, 9:48am

Erm, apologies, you’d circled the thing most people ask about - if you mean you never get an entries in the log underneath, then there is something preventing the device from matching EUI’s etc at the Network Server level so it can’t pass it on to the application console.

How are you joining, what is the device, the stack, it’s version and can you check the EUI’s / keys.

ebenl · September 15, 2022, 10:15am

Hi Nick, below is a summary on the configuration :

Gateway

RAK7268
Configuration Mode : Basics Station
Gateway EUI : AC1F09FFFE0606D0
Server setup : LNS Server wss://eu1.cloud.thethings.network
Firmware : WisGateOS 1.3.4

Lora Node

RAK3172
Firmware : 3.2.0-p2_22q1_final.87
Join mode : OTAA
Class A
DEVEUI=70B3D57ED00553C8
APPEUI=0000000000000000
APPKEY=D0…74

Application

AppEUI = 0000000000000000
DevEUI = 70B3D57ED00553C8
AppKey = D0…74
LoRaWAN version : LoRaWAN Specification 1.0.3

I join network by sending AT command AT+JOIN=1:0:10:8, then after joined ack recevied I send test data AT+SEND=1:0167FFD7

ebenl · September 15, 2022, 10:41am

Hi Nick, I see live data now showing in the console

Did you perhaps change anything? Awesome to see the data!