# V3 console Live Data filtering **URL:** \ **Category:** V3 Console\ **Created:** [February 2, 2021, 5:25pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671 "2021-02-02T17:25:34Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![jezd](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/jezd/32/46461_2.png) [@jezd](https://www.thethingsnetwork.org/forum/u/jezd)\ **Post date:** [February 2, 2021, 5:25pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/1 "2021-02-02T17:25:34Z")
After finally getting my hands on V3 console its a major improvement - congrats to all those who have worked on it. The live data feed whist very comprehensive could do with a few mods in my view as it feels like information overload. Filtering is the top issue, I’ve seen discussion on github but that seems to have closed down so my question would be is filtering a missing feature? This could be built into the console functionality for example. Also colour highlighting of key values would be very helpful, just one example if the FCnt value was say on a green background. I am wondering if there is any option here for customising the live view data stream, perhaps some CSS we can edit to change the look and feel? Just bouncing ideas around (if I’ve duplicated something let me know, I did search). ---
**Author:** ![cslorabox](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/cslorabox/32/35697_2.png) [@cslorabox](https://www.thethingsnetwork.org/forum/u/cslorabox)\ **Post date:** [February 2, 2021, 6:32pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/2 "2021-02-02T18:32:00Z")
> [@jezd](#): > > Filtering is the top issue, I’ve seen discussion on github but that seems to have closed down so my question would be is filtering a missing feature? What do you want to filter? For the most part, my understanding is that the console is for quick debug; to really work with your data, you probably want to be routing things to a data platform of your choosing. For that matter, I end up doing a lot with `tail` and `grep` working on simple subscription logs… ---
**Author:** ![kersing](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/kersing/32/44777_2.png) [@kersing](https://www.thethingsnetwork.org/forum/u/kersing)\ **Post date:** [February 2, 2021, 7:18pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/3 "2021-02-02T19:18:53Z")
> [@cslorabox](#): > > What do you want to filter? In the console for a busy gateway it is useful to be able to filter an address and types of traffic (uplink/downlink/activation). Some sites have too much traffic to be able to find the information by just looking at the console. ---
**Author:** ![jezd](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/jezd/32/46461_2.png) [@jezd](https://www.thethingsnetwork.org/forum/u/jezd)\ **Post date:** [February 3, 2021, 2:48pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/4 "2021-02-03T14:48:01Z")
V2 console has data filters and they are used to help diagnose issues, V3 console provides even more granular information (which is great) so filters would be very useful - a single uplink from one device can now involve 10+ lines in the live data window, without filtering application live data view will be pointless given the amount of data rushing past. Also noted that V3 console live data seems to have a timeout, this is new and a bit frustrating. ---
**Author:** ![jpmeijers](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/jpmeijers/32/44875_2.png) [@jpmeijers](https://www.thethingsnetwork.org/forum/u/jpmeijers)\ **Post date:** [February 3, 2021, 3:42pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/5 "2021-02-03T15:42:54Z")
Yes a filter for the Live Data feed would be very useful. Currently all events are shows, making it difficult to spot the actual uplink message with payload. ---
**Author:** ![cslorabox](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/cslorabox/32/35697_2.png) [@cslorabox](https://www.thethingsnetwork.org/forum/u/cslorabox)\ **Post date:** [February 3, 2021, 4:46pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/6 "2021-02-03T16:46:41Z")
> [@jpmeijers](#): > > Currently all events are shows, making it difficult to spot the actual uplink message with payload. Are you looking at a device view or a gateway view? I’m not meaning that there’s never a role for filtering a raw feed, but in general the console isn’t typically where you’d want to look for application-level device data. Especially you wouldn’t look at a gateway view, as the device data is still encrypted there. In terms of debugging devices from a gateway feed also seeing others, one of the challenges for filtering is what you’d be looking for. Device address gets changed each join accept, and the code presenting a gateway view may not really have access to the information needed to turn that back into a Device EUI. That EUI is present in join requests, but then vanishes from the resulting packets within a session… When doing device development, hacking a packet forwarder or system it’s running on to twin the feed to a textual logfile you can grep might not be a bad idea. ---
**Author:** ![jezd](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/jezd/32/46461_2.png) [@jezd](https://www.thethingsnetwork.org/forum/u/jezd)\ **Post date:** [February 3, 2021, 5:10pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/7 "2021-02-03T17:10:58Z")
You keep mentioning gateway view, no not yet, not even looked at moving gateways to V3 yet - this is simply about the device live data and application live data feeds. ---
**Author:** ![cslorabox](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/cslorabox/32/35697_2.png) [@cslorabox](https://www.thethingsnetwork.org/forum/u/cslorabox)\ **Post date:** [February 3, 2021, 5:12pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/8 "2021-02-03T17:12:29Z")
> [@jezd](#): > > this is simply about the device live data and application live data feeds. Surely that shows you the most recent packet that’s come in while you are watching. For any more historic extraction of data, you should be pushing it to an appropriate data platform which actually retains data, not using the console. What it comes down to is that you seem to be trying to use this page for a purpose it’s not really intended to fill. ---
**Author:** ![jezd](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/jezd/32/46461_2.png) [@jezd](https://www.thethingsnetwork.org/forum/u/jezd)\ **Post date:** [February 3, 2021, 5:26pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/9 "2021-02-03T17:26:34Z")
Exactly what am I doing wrong that your not happy with? It’s not hard this, the filters exist in V2 and I use them in diagnosing issues in both applications and devices (I’m guessing others do too), I just thought they would be of value in V3 given the information in the new live view is even greater. You seems to be suggesting I am doing it wrong, I’m just using the live data to help me that’s all and I can see it being of no use once large numbers of devices populate an application - in fact the application live date view will be pointless without filtering as the data will volumes will be too high. ---
**Author:** ![cslorabox](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/cslorabox/32/35697_2.png) [@cslorabox](https://www.thethingsnetwork.org/forum/u/cslorabox)\ **Post date:** [February 3, 2021, 5:27pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/10 "2021-02-03T17:27:21Z")
> [@jezd](#): > > in fact the application live date view will be pointless without filtering as data will volumes will be too high. Look at the device feed, not the application one You’re indeed right, there isn’t much use for an “application” web view - as I was explaining earlier, you really should be routing your data into a data platform which you can query in whatever way you desire, and which will show you _historic_ data in a way that the web view cannot. ---
**Author:** ![jezd](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/jezd/32/46461_2.png) [@jezd](https://www.thethingsnetwork.org/forum/u/jezd)\ **Post date:** [February 6, 2021, 3:13pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/11 "2021-02-06T15:13:59Z")
Web browser scroll bars in console are a nightmare in V3 compared to V2 console, must be set at 25% of those is V2 (ie normal) ---
**Author:** ![mathieu\_pouillot](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/mathieu_pouillot/32/35697_2.png) [@mathieu\_pouillot](https://www.thethingsnetwork.org/forum/u/mathieu_pouillot)\ **Post date:** [May 18, 2021, 10:36am UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/13 "2021-05-18T10:36:23Z")
+1 to the filter option. It is a real lack on this release to debug quickly just to show simple uplink, downlink, join,… best regards, Mathieu ---
**Author:** ![descartes](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/descartes/32/47220_2.png) [@descartes](https://www.thethingsnetwork.org/forum/u/descartes)\ **Post date:** [May 18, 2021, 10:39am UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/14 "2021-05-18T10:39:44Z")
I’m working on live data tool to help with this - code named FireHose (as in, try drinking from one). I’ll be needing some testers shortly, however they will have to be able read instructions and deal with a grumpy sarcastic old developer. ---
**Author:** ![Jeff-UK](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/jeff-uk/32/35697_2.png) [@Jeff-UK](https://www.thethingsnetwork.org/forum/u/Jeff-UK)\ **Post date:** [May 18, 2021, 10:44am UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/15 "2021-05-18T10:44:27Z")
> [@descartes](#): > > however they will have to be able read instruction Unlike the current tester 😉 ---
**Author:** ![descartes](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/descartes/32/47220_2.png) [@descartes](https://www.thethingsnetwork.org/forum/u/descartes)\ **Post date:** [May 18, 2021, 10:45am UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/16 "2021-05-18T10:45:37Z")
But he does cope well with the grumpy sarcastic old developer! Birds of a feather? ---
**Author:** ![kschiffer](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/kschiffer/32/46792_2.png) [@kschiffer](https://www.thethingsnetwork.org/forum/u/kschiffer)\ **Post date:** [May 18, 2021, 11:34am UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/17 "2021-05-18T11:34:28Z")
Hey everyone! Note that filtering out verbose events by default is a feature coming up in the next release which will be deployed shortly thereafter. It is a first step towards improving the usability of the event views in the Console (as in taming the firehose 😅). See also: > > > \#### Summary > > This PR will apply a toggle-able verbosity filter to the event s…tream which will by default filter out all events that are not typically of direct interest. > > !\[\](https://user-images.githubusercontent.com/5710611/117699085-b4cd6c80-b1c4-11eb-8bbf-5abb8c010d8c.png) > > References #2231 > > With verbose mode deactivated (default) only the following events will be ingested: > > \`\`\`javascript > export const EVENT\_VERBOSE\_FILTERS = \[ > 'as.\*.drop', > 'as.down.data.forward', > 'as.up.data.forward', > 'gs.down.send', > 'gs.gateway.connect', > 'gs.gateway.disconnect', > 'gs.status.receive', > 'gs.up.receive', > 'js.join.accept', > 'js.join.reject', > 'ns.mac.\*.answer.reject', > '\*.warning', > '\*.fail', > 'organization.\*', > 'user.\*', > 'gateway.\*', > 'application.\*', > 'end\_device.\*', > 'client.\*', > 'oauth.\*', > \] > \`\`\` > > \#### Changes > > \* Add a constant module for event filters > \* Update store logic for applying filters to event streams > \* Update the \`\\` component to enable filtering out verbose events > \* Update entity event containers accordingly > > \#### Testing > > Manual testing in the staging environment > > \##### Regressions > > This could affect event ingestion in the Console > > \#### Notes for Reviewers > > \* I did not add the filter for organization events, since the filter would have no effect. For completeness I added the redux primitives for organizations though. > \* I have implemented this filter at the ingestion level within the reducer, meaning that filtered events will not be added to the store. This means that historical verbose events can not be toggled, since they were already discarded. I've decided to go this route to also lift some weight on the whole event store logic (see also #2887). With this implementation it will be possible to examine events for entities with a higher event frequency. > \* Later on, other filter could be applied on top of this but filtered out on the component level so that the visibility can be toggled for past events as well. > > \#### Checklist > > \* \[x\] Scope: The referenced issue is addressed, there are no unrelated changes. > \* \[x\] Compatibility: The changes are backwards compatible with existing API, storage, configuration and CLI, according to the compatibility commitments in \`README.md\` for the chosen target branch. > \* \[x\] Documentation: Relevant documentation is added or updated. > \* \[x\] Changelog: Significant features, behavior changes, deprecations and fixes are added to \`CHANGELOG.md\`. > \* \[x\] Commits: Commit messages follow guidelines in \`CONTRIBUTING.md\`, there are no fixup commits left. ---
**Author:** ![descartes](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/descartes/32/47220_2.png) [@descartes](https://www.thethingsnetwork.org/forum/u/descartes)\ **Post date:** [May 18, 2021, 11:52am UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/18 "2021-05-18T11:52:03Z")
Cool - your’s is Lego Technic - mine is very much Lego Duplo. @kschiffer, assuming you are at the heart of the Matrix, can you explain the correlation ids please - what are they, how might we use them and more importantly, particularly for mobile apps, can we turn them off? ---
**Author:** ![kschiffer](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/kschiffer/32/46792_2.png) [@kschiffer](https://www.thethingsnetwork.org/forum/u/kschiffer)\ **Post date:** [May 18, 2021, 12:03pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/19 "2021-05-18T12:03:01Z")
> Cool - your’s is Lego Technic - mine is very much Lego Duplo. Not sure if that is good or bad. You mean that you would like to have even less data displayed (even more data filtered out)? Correlation IDs are, as the name suggests, IDs that allow you to link an event to other event based on a specific topic. E.g. for gateways, you’ll see a correlation ID for the connection, allowing you to identify all gateway events that are related to this particular gateway connection. Another topic would be e.g. an uplink. You will see that all gateway events relating to the same uplink will share the same correlation ID starting with `gs:uplink:`. In the future we will use these IDs in the Console to highlight related events. I’m not exactly sure what you mean by “turning them off”. If you mean to make the event stream omit the correlation IDs (or any other event property), then yes, something like this is in consideration. ---
**Author:** ![descartes](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/descartes/32/47220_2.png) [@descartes](https://www.thethingsnetwork.org/forum/u/descartes)\ **Post date:** [May 18, 2021, 12:24pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/20 "2021-05-18T12:24:42Z")
> [@kschiffer](#): > > Not sure if that is good or bad. You mean that you would like to have even less data displayed (even more data filtered out)? Both - I like yours for the detail but when I’m running round the garden pouring water on to sensors, I’d rather have a really simple display that just shows me that an uplink was triggered which means the alarm was triggered. I’ve got something that replicates the v2 traffic view without the expanding detail working, just needs a bit more tidy up before beta-testing. Plus mobile needs the bare minimum unless you have an A2 side iPad Air. But when it gets gritty, return to “The Source” - the official console - what would be considered “The Truth”. I’d not worry about the many variants of live data that people could aspire to - the API’s are there for people to roll their own and it means the wife is happy because I’m fully occupied. Understand the correlation id’s now - very good - but mostly they are wasted bytes for the average console view - if I run out of things to do I could get some API feeds running and have different screens linking the entries via the correlation-ids - probably need to be a glass freestanding monitor with gesture control so I can swipe widgets around. ---
**Author:** ![kschiffer](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/kschiffer/32/46792_2.png) [@kschiffer](https://www.thethingsnetwork.org/forum/u/kschiffer)\ **Post date:** [May 18, 2021, 12:33pm UTC](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671/21 "2021-05-18T12:33:31Z")
I see. Yeah indeed you’re always free to come up with your own interface that suits your specific needs (which is part of the power of TTS). Obviously though in the Console we try to come up with a design that suits as many needs as possible. As is known, we’re not quite there yet, but improvements will come in steadily now. Btw, regarding mobile, the TTS Console is fully responsive and also the event views are optimized for mobile screensizes (obviously with some limitations). We will flesh this out further with the ability to hide columns as well, but it can already be used pretty smoothly (imho). ![image](https://www.thethingsnetwork.org/forum/uploads/default/original/3X/6/4/6438070a5d848d15635051ebcbb870826440b2f3.png) [Next page](https://www.thethingsnetwork.org/forum/t/v3-console-live-data-filtering/43671.md?page=2)