Plotting Large Datasets

## Data, Where You At?

<p class="fragment" data-fragment-index="6" style="margin-right: auto; width: max-content;"><code>for _ in range(270_000):</code></p>

<div class="dialogue-left">
                  Hey data field, what's your content?
                </div>

<div class="dialogue-right fragment" data-fragment-index="1">
                  I have a thing here, let me get it out of this dict for you!
                </div>

<div class="dialogue-left fragment" data-fragment-index="4">
                  Not cool, please make it <code>other_data_type</code>
                </div>

<div class="dialogue-right fragment" data-fragment-index="5">
                  [numbercrunch] Sure, here you go!
                </div>

Recommendation

Disentangling front-end and back-end

Easier development, testing, debugging
Easier data wrangling with a DB interface
Better performance when retrieving data from DB

Data Types

Benchmarks

timeit, best of 50 runs
Data on RAMdisk to counter caching variability

Code Investigation:
Expensive Operations

type checking
nested dicts
append to list

Data Wrangling

(spinedb_api: by ID)

Recommendation

Typing and Checking during Data Creation

Separation of responsibilities

Maintainable code

Less computations at run-time
Clearer data structure

Data Structure

Dict Nesting

Queryable JSON BLOBs

{
  "data": {
    "2000-01-01T00:00:00.0": 90.0,
    "2000-01-01T01:00:00.0": 91.0,
    "2000-01-01T02:00:00.0": 93.0
  },
  "index": {
    "ignore_year": false,
    "repeat": false
  }
}

Queryable JSON BLOBs

{
  "data": [
      {"t": "2000-01-01T00:00:00.0", "v": 90.0},
      {"t": "2000-01-01T01:00:00.0", "v": 91.0},
      {"t": "2000-01-01T02:00:00.0", "v": 93.0}
  ],
  "index": {
    "ignore_year": false,
    "repeat": false
  }
}

Q. JSON BLOBs: Performance

N.B.: adbc now compatible with our data format! 🥳

Discussing Parquet

Recommendation

Using a Queryable Data Structure

Speed!
Use DB for searching, filtering, etc.
Separation of responsibilities (deja vu)

Maintainable code

Plotting Large Datasets

Recommendation

Disentangling front-end and back-end

Data Types

Benchmarks

Code Investigation:Expensive Operations

Data Wrangling

Recommendation

Typing and Checking during Data Creation

Data Structure

Dict Nesting

Queryable JSON BLOBs

Queryable JSON BLOBs

Q. JSON BLOBs: Performance

Discussing Parquet

Recommendation

Using a Queryable Data Structure

Questions?

Code Investigation:
Expensive Operations