Db v4 schema layer - #2658
Open
arienandalibi wants to merge 40 commits into
Open
Conversation
…global property id i corresponding to the segment. Persisted to disk. In-memory part grows incrementally.
…ow) of properties present in each (segment, layer) to efficiently retrieve the lists of relevant properties. We union over all segments in the storage. We also filter out props/metadata that aren't present now instead of all globally registered properties.
… for properties and one for metadata in Meta. Wired it in so that layers where we know the property isn't present are skipped.
…und in this layer. Added these for the node storage and edge storage. Skip layers in tprop_iter_layers, which is used in time semantics.
…es. These were used to track property presence per-(segment, layer) on disk and in-memory. Now, we only keep track of property presence per layer. We skip layers where we know the property was added, but that's it.
… previous layer property schemas (property presence bitsets) to avoid work. All of it was redundant and didn't save anything. Removed functions that pass these `LayerPropSchema`s around.
…pe, dst_node_type). Introduce limits for max number of edges/values in properties and edges.
# Conflicts: # raphtory-storage/src/graph/nodes/node_storage_ops.rs # raphtory/src/db/graph/views/filter/model/degree_filter.rs
…g all properties. As soon as a property key has more than ENUM_BOUNDARY values, we skip property value collection.
…ies and metadata. This is held by GraphWithVectors(Inner). It gets passed to GqlGraph for base/unfiltered (e.g. materialized) graphs, which passes them to EdgeSchema by Arc.
…String, String) type alias. This allows us to pass references to the key easily. Faster on cache lookup
# Conflicts: # raphtory-storage/src/graph/nodes/node_storage_ops.rs
# Conflicts: # raphtory-graphql/src/data.rs # raphtory-graphql/src/graph.rs # raphtory-storage/src/graph/nodes/node_storage_ops.rs
…es and metadata per layer. We use the maintained property presence bitsets to do this
… rid of NodeSchemaKey. Wire the cache through in the schemas.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.
| Benchmark suite | Current: e33298d | Previous: 9823ef7 | Ratio |
|---|---|---|---|
lotr_graph/num_edges |
2 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph/num_nodes |
3 ns/iter (± 0) |
1 ns/iter (± 0) |
3 |
lotr_graph/graph_latest |
2 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph_materialise/materialize |
5368222 ns/iter (± 13405) |
1564816 ns/iter (± 35303) |
3.43 |
lotr_graph_window_100_materialise/materialize |
5408279 ns/iter (± 27426) |
1669150 ns/iter (± 10700) |
3.24 |
lotr_graph_window_10_materialise/materialize |
2211315 ns/iter (± 14340) |
971980 ns/iter (± 4278) |
2.28 |
lotr_graph_subgraph_10pc/num_nodes |
13 ns/iter (± 0) |
4 ns/iter (± 0) |
3.25 |
lotr_graph_subgraph_10pc_materialise/materialize |
1798460 ns/iter (± 101028) |
334634 ns/iter (± 1287) |
5.37 |
lotr_graph_subgraph_10pc_windowed_materialise/materialize |
1001215 ns/iter (± 44248) |
230399 ns/iter (± 2617) |
4.35 |
lotr_graph_window_50_layered/has_node_existing |
261 ns/iter (± 14) |
129 ns/iter (± 12) |
2.02 |
lotr_graph_window_50_layered_materialise/materialize |
19471688 ns/iter (± 50779) |
3488825 ns/iter (± 24948) |
5.58 |
lotr_graph_persistent_window_50_layered/num_edges_temporal |
397937 ns/iter (± 31285) |
192686 ns/iter (± 1569) |
2.07 |
lotr_graph_persistent_window_50_layered_materialise/materialize |
33377573 ns/iter (± 2445220) |
5298035 ns/iter (± 147912) |
6.30 |
This comment was automatically generated by workflow using github-action-benchmark.
…e bitset to integration-tests (in pometry-storage) instead of raphtory-tests (in Raphtory). Fixed queries and responses to match new schemas.
arienandalibi
marked this pull request as ready for review
July 8, 2026 08:15
fabubaker
self-requested a review
July 8, 2026 16:08
fabianmurariu
requested changes
Jul 27, 2026
fabianmurariu
left a comment
fabianmurariu
left a comment
Collaborator
There was a problem hiding this comment.
Maybe we can avoid some clones?
| self.graph_stats.update_time(t.t()); | ||
| // Update the per-layer property presence bitset in Meta. | ||
| // `.inspect` runs once per emitted item as the iterator is consumed in `insert_edge_internal` | ||
| let meta = self.writer.edge_meta().clone(); |
Collaborator
There was a problem hiding this comment.
This is going to light up in bulk loading quite badly, we need a clone free variant
| self.layer_prop_presence | ||
| .read_recursive() | ||
| .get(layer_id.0) | ||
| .and_then(|row| row.get(prop_id)) |
| key.to_string(), | ||
| mapper | ||
| .get_dtype(id) | ||
| .expect("type for internal id should always exist") |
Collaborator
There was a problem hiding this comment.
unwrap_or_else with a panic inside is a bit better
# Conflicts: # db4-storage/src/pages/node_page/writer.rs
…rty presence bitsets. Instead, we add a function on SegmentContainer which can do it with borrowing while avoiding borrowing issues (mut and immut).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adding a bitset (Vec) to keep track of which properties exist in which layers. Use this to optimize iterators (especially relevant when using disk storage). Also optimized the generation of schemas (node schema, edge schema, ...) by iterating edges only once and implementing a cache for already generated schemas.
Why are the changes needed?
Optimizations can be made to improve UI experience and disk accesses.
Does this PR introduce any user-facing change? If yes is this documented?
It shouldn't
How was this patch tested?
Will need to be tested using the UI
Are there any further changes required?
There shouldn't be