📕 Node [[dashboard]]
📄 dashboard.md by @karlicoss

Table of Contents

Dashboard (not a very unique name, I know) is a set of tools to visualize personal data on a, well, some sort of dashboard.

I’m gradually publishing it here and got some old screenshots here.

Ideally I’d like to reuse existing libraries and other people’s projects to the maximum extent possible, so we could join forces.

related

used to visualize #qs data [[qs]]

very similar to #timeline and #memex [[timeline]] [[memex]]

I haven’t quite figured out the difference as well yet. Likely will merge in a single tool.

#hpi provides data for my dashboard [[hpi]]

Although in theory you can plug in any Pandas dataframes in it (assuming they fit the schema).

* motivation

  • same motivation as for quantified self
  • a ‘research’ goal: figuring out the patterns and common routines for processing and preparing this data for visualizing.

why not use something existing?

  • low-code/no-code ‘platforms’ I’ve seen aren’t enough to deal with noisy data or novel usecases.

    • filtering outliers, correcting data, etc., is mipossible in a no-code environment
    • something like seasonality detection would be impossible without a real programming language

  • interoperability: if you want some data source that the platform doesn’t support, too bad.

figuring out core patterns for manipulating and representing this data

Many things are very tedious to redo and reinvent from scratch every time

  • #datetime handling

    • missing dates (e.g. parsing errors)
    • some data sources hane timezones, some don’t.
      Some are a mixture, which is even worse. E.g. depending on the language you can’t sort the dates with mixed timezones.

  • #errors handling
    Quantified self data is often fuzzy/incomplete etc. If manual inputs are involved, parsing errors are inevitable.
    While in most software it’s reasonable to crash on the first error, this would be very annoying if your dashboard breaks of one broken data point!
    On the other hand, you don’t want to filter out errors either, since you’d never know they were present in the first place.

    So it needs to be something in between, a pattern for propagating errors as far as possible, and handling them gracefully during the visualization phase.

  • detecting holidays
    E.g. for almost all data it makes sense to differentiate/correlate against days off work

  • geographical/location data

automatically detecting interesting correlations between data

[2019-08-15] wonder if massive dataframe with literally everything could help

maybe just go over all series? and then generate all pairwise correlations and sort by correlation coefficient + R2? some sort of pareto surface?

what I want to figure out:

very similar to https://github.com/KrauseFx/FxLifeSheet#questions-to-answer

are ‘sleep phases’ a thing (or at least the way they are measured in conventional tackers)? how do they impact my waking up, alertness etc? [[sleep]]

[2020-10-27] calculating custom exercise stats

e.g. I have a few traffic lights on my way that might affect my speed/cadence data, and I might want to filter it out

custom/personalized data processing

splitting exercise into cardio and non-cardio

  • with cardio (at least, endurance styl) there is a somewhat convincing metric – number of heartbeats, which is basically the same as calories burnt?
  • with strength training this is something very hard to measure and not sure if makes sense at all?

demo of errors: typical mistake is confuising wlog and weight tags (weight gets unrecognized as exercise) [[dashboard]]

* inspiration

[2018-06-06] andreilyskov: qs dashboard example [[qs]]

  • "so if you want to collaborate feel free to email me"

[2019-04-06] interesting dashboards: spotify, writing and reading. otherwise not that much stuff

[2020-08-22] ok seems that they use similar architecture to mine https://forum.quantifiedself.com/t/personal-dashboards-for-self-tracking-data/8202/4

[2020-08-22] Flow Dashboard

ok, looks kinda slick..
open source: https://github.com/onejgordon/flow-dashboard 

Public Github commits
Google Fit - track any activity durations by keyword
Evernote - pull excerpts from specified notebooks
Pocket - Sync stored articles & add notes
Goodreads - Sync currently reading shelf
Track any abstract data via REST API

[2021-02-06] ok, rest api thing is cool, should try connecting it to #hpi [[hpi]]

[2019-11-24] heedy/heedy: An Open-Source Platform for Quantified Self & IoT

Heedy is an open-source aggregator built for storage and analysis of your personal data.
It provides a powerful plugin system, allowing easy integration with various services, as well as deep extensibility.

Extensible: Even a system with fantastic visualizations and powerful analysis has limited utility.
This is because it can only show what the original authors assumed would be useful. Heedy offers a powerful plugin system - plugin writers can add new integrations, plots, or even modify core functionality with a few lines of python or javascript.
A registry is planned, so that users can install plugins with the click of a button.
  • [2020-08-23] interesing, automatic jupyter notebook connection??
  • [2021-02-06] yeah looks like very much what I want
  • [2021-04-27] hmm
    seems that fitbit integration is the only one present so far
    however after giving it a go, it restarted a couple of times, and ended up with a couple of zombie processes
    couldn’t restart it without completely tearing down the docker container :(
    Overall not sure where it all fits: there are lots of repositories and not many concrete usecases https://github.com/heedy

[2019-05-21] quelf/report.Rmd at master · JakobGM/quelf

jupyter notebooks 

Sleep data [sleep as android, sleepcycle]
Study hours [toggl]
Exercise [runkeeper, strong]
Programming [wakatime]

[2019-09-22] markwk/qsledger: Quantified Self Personal Data Aggregator and Data Analysis [[qs]] [[dataliberation]]

interesting, but data exports are not very reusable

[2019-09-28] I guess once I export everything could send a link to that guy?

[2020-09-03] https://github.com/KrauseFx/FxLifeSheet

the dashboard is currently hard-coded in Google Data Studio

[2020-08-22] some good dashboards https://forum.quantifiedself.com/t/personal-dashboards-for-self-tracking-data/8202/15

workouts, weight, sleep, hr

Juno’s Personal Data Exploratory https://exploratory.openhumans.org/notebooks [[pkm]] [[qs]]

[2020-07-02] jeffshek/betterself: Your body’s dashboard.

https://github.com/jeffshek/open

[2020-08-24] demo https://app.betterself.io

  • table with notes is kinda nice?
    eh. would be nice to have something automatic for dataframes

[2019-01-13] Sacha Chua: quantified awesome [[qs]]

realtime dashboard

[2019-04-06] ok, looks pretty clean, but might require some tinkering

[2021-02-06] also pretty much only time tracking?

[2020-09-10] quantifiedbob/bob-body-composition-viz: Jupyter Notebook vizualizing 10+ years of my body composition data

ok, but just a plot

[2020-04-04] ammanvedi/quantified-self-server: Aggregate data about myself (workouts, blog posts, music listening history) into a graphql API

Blog posts (markdown from github)
Workout data (Strava API)
Recent music (Tidal API)
into one graphql API that can be called from my personal site.

[2019-04-19] markwk: How to Create a Time Tracking Dashboard using RescueTime, IFTTT and Google Sheets [[qs]]

ok, but too low code for me

[2019-01-08] Personal Dashboard /r/QuantifiedSelf

https://i.redd.it/ufpm4s9k0o821.png

[2019-04-11] it is not very elaborate though…

minor ideas

keeping query in the address string is pretty clever… can bookmark and easily restart it! [[datasette]]

[2020-08-22] event chart https://forum.quantifiedself.com/t/best-examples-of-quanitified-self-dashboards/4590/9

use tabs within tabs? kinda like in garmin https://connect.garmin.com/modern/activities

[2020-10-27] Garmin Connect

pretty nice clean visualizations, I need to borrow these I guess

[2021-01-11] yihong0618/runningpage: Make your own running home page

* implementation

How to actuallly implement it, which software/tools/libraries to use.

(meta?)goals

  • interact with the backend (so it’s possible to reload newer data)
    plotly can do it, bokeh can as well
    • ideally, autorefresh?
  • remember the state (e.g. various panes on/off etc)
  • be hackable
    • via js
    • autoreload the code for faster iteradion?
  • be able to derive new data on the fly? not sure how to achieve this in the browser
  • have a jupyter interface for better interactivity? ideally, ‘edit’ any tab as a jupyter notebook

keep it modular, I guess

have a core subpackages and various tabs… not sure how to refactor them out later

[2020-09-01] similarly to HPI, it’s more of a demonstration how it can be done. Ideally people would be able to reuse core and build their own dashboards

it’s very important to benefit from the existing data science infrastructure as much as possible. jupyter, pandas, etc.

[2020-12-10] panel · PyPI

https://panel.holoviz.org/ 

A high-level app and dashboarding solution for Python
Panel works with visualizations from Bokeh, Matplotlib, HoloViews, and many other Python plotting libraries

ok, this is super promising: 

Panel can also be used with the separate Param project to create interactively configurable objects with or without associated visualizations, in a fully declarative way. With this approach, you declare your configurable object using the pure-Python, zero-dependency param library, annotating your code with parameter ranges, documentation, and dependencies between parameters and your code. Using this information, you can make all of your domain-specific code be optionally configurable in a GUI, with optional visual displays and debugging information if you like, all with just a few lines of declarations. With this approach, you don’t ever have to decide whether your code will eventually be used in a notebook, in a GUI app, or completely behind the scenes in batch processing, servers, or reports – the same code can support all of these cases equally well, once you declare the associated parameters and constraints.

[2020-12-07] oschuett/appmode: A Jupyter extensions that turns notebooks into web applications.

A Jupyter extensions that turns notebooks into web applications.

[2021-02-06] ok, nice, has ‘edit app’ button which opens jupyter to edit??

[2020-08-23] ok, #jupyter is the ultimate ad-hoc solution. it preserves state, can be exposed as a frontend and allows for python code

bokeh vs plotly dash

there are couple of example apps here
https://www.sicara.ai/blog/2018-01-30-bokeh-dash-best-dashboard-framework-python

bokeh

  • seems snappier?

plotly:

  • possibly better html elements support?

[2020-09-03] compromise: keep it as framework independent and decoupled as possible. Support both frameworks!

basically have a core library that’s imported in all notebooks. move stuff to core as long as possible, but allow quick button to mess with the data in notebook [[toblog]]

try jupyterlab

I think dashboard needs to preserve state somehow, it’s annoying to edit code every time.. wonder if dash supports it?

use multiprocessing…

pretty sure pandas frames can be easily serialised?

[2020-09-10] Top 50 matplotlib Visualizations - The Master Plots (w/ Full Python Code) | ML+

39. Time Series Decomposition Plot

ok, this is pretty intersting
re: autocorrelation – what was up with the ‘blue region’, significance thing??
also good: Seasonal Plot
would be initersting to quickly toggle it in dashboard?

grafana could simply be one of the interfaces

[2021-02-17] apache/superset: Apache Superset is a Data Visualization and Data Exploration Platform [[dashboard]] [[hpi]]

A modern, enterprise-ready business intelligence web application.

whoa looks interesting…

  • [2021-04-26] trying sqlite connection

Ugh. it didn’t really work – seems to be in the process of deprecation
https://github.com/apache/superset/issues/9748
and still, getting some weird error regardless whether I mount database as read only or not.. and no other logs 

2021-04-26 14:19:42,813:WARNING:superset.views.base:[SupersetError(message='(sqlite3.OperationalError) unable to open database file\n(Background on this error at: http://sqlalche.me/e/13/e3q8)', error_type=<SupersetErrorType.GENERIC_DB_ENGINE_ERROR: 'GENERIC_DB_ENGINE_ERROR'>, level=<ErrorLevel.ERROR: 'error'>, extra={'engine_name': 'SQLite', 'issue_codes': [{'code': 1002, 'message': 'Issue 1002 - The database returned an unexpected error.'}]})]
  • [2021-04-26] right, managed to run against postgre..

a bit finicky overall…
I guess nice that it’s possible to separate queries, charts and dashboards…
so one can reuse them in different contexts/for different visualizations

  • [2021-04-26]
    for fuck’s sake, struggling with creating a ‘virtual’ view with calculated columns…
    getting an error like 

    Error: column "_cachew_union_repr_measurement_temp" does not exist

  • [2021-04-26] seems possible to share queries?

* sleep dashboard/experiments [[sleep]]

Sleep is one of the biggest things I wanna figure out, and I have quite a few ideas of what to test.

Maybe run pca for my sleep data? [[sleep]] [[qs]] [[ml]]

figure out main questions I wanna figure out [[qs]] [[sleep]]

e.g. ‘what is the best time for me to go asleep’?

‘what is the best sleep duration?’

[2019-08-21] Respiratory rate - Wikipedia [[sleep]]

https://en.m.wikipedia.org/wiki/Respiratory_rate

⸎ (text) last updated 2021-05-16 01:16:22+02:00
📄 dashboard.md by @an_agora@twitter.com
⸎ (text) last updated 2022-03-13 23:28:33+01:00

Loading pushes...

Rendering context...

📕 Node [[2004 07 10 dashboard apps multiplying]] perhaps related
📕 Node [[agora dashboard]] perhaps related
📕 Node [[dashboards]] perhaps related
📕 Node [[week5 us economic dashboard in python lab]] perhaps related