Remote py apis - #2675
Draft
shivamka1 wants to merge 114 commits into
Draft
Conversation
…ree read wired end-to-end
…ata, update_metadata, delete_edge) through Transport
…date_metadata) through Transport
…tadata) through Transport
…aph.node/edge to Rust methods (fixes view-chain preservation in Python)
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.
| Benchmark suite | Current: 2b624f9 | Previous: 9823ef7 | Ratio |
|---|---|---|---|
lotr_graph/num_edges |
4 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph/num_nodes |
5 ns/iter (± 0) |
1 ns/iter (± 0) |
5 |
lotr_graph/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph/graph_latest |
3 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph_materialise/materialize |
8050631 ns/iter (± 40172) |
1564816 ns/iter (± 35303) |
5.14 |
lotr_graph_window_100/num_nodes |
15 ns/iter (± 0) |
5 ns/iter (± 0) |
3 |
lotr_graph_window_100_materialise/materialize |
8089774 ns/iter (± 64519) |
1669150 ns/iter (± 10700) |
4.85 |
lotr_graph_window_10/has_node_existing |
143 ns/iter (± 9) |
62 ns/iter (± 11) |
2.31 |
lotr_graph_window_10_materialise/materialize |
3345959 ns/iter (± 47014) |
971980 ns/iter (± 4278) |
3.44 |
lotr_graph_subgraph_10pc/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_subgraph_10pc_materialise/materialize |
2009904 ns/iter (± 22468) |
334634 ns/iter (± 1287) |
6.01 |
lotr_graph_subgraph_10pc_windowed/has_node_existing |
149 ns/iter (± 9) |
62 ns/iter (± 14) |
2.40 |
lotr_graph_subgraph_10pc_windowed_materialise/materialize |
1217703 ns/iter (± 11569) |
230399 ns/iter (± 2617) |
5.29 |
lotr_graph_window_50_layered/num_edges_temporal |
155717 ns/iter (± 2232) |
70121 ns/iter (± 7586) |
2.22 |
lotr_graph_window_50_layered/has_node_existing |
390 ns/iter (± 24) |
129 ns/iter (± 12) |
3.02 |
lotr_graph_window_50_layered/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_window_50_layered/graph_latest |
85415 ns/iter (± 2064) |
36649 ns/iter (± 916) |
2.33 |
lotr_graph_window_50_layered_materialise/materialize |
30515575 ns/iter (± 95819) |
3488825 ns/iter (± 24948) |
8.75 |
lotr_graph_persistent_window_50_layered/num_edges_temporal |
646047 ns/iter (± 24446) |
192686 ns/iter (± 1569) |
3.35 |
lotr_graph_persistent_window_50_layered/has_node_existing |
442 ns/iter (± 435) |
174 ns/iter (± 83) |
2.54 |
lotr_graph_persistent_window_50_layered/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_persistent_window_50_layered/iterate_exploded_edges |
3480647 ns/iter (± 10244) |
1659940 ns/iter (± 19402) |
2.10 |
lotr_graph_persistent_window_50_layered/graph_latest |
137704 ns/iter (± 4154) |
57549 ns/iter (± 4809) |
2.39 |
lotr_graph_persistent_window_50_layered_materialise/materialize |
53184087 ns/iter (± 282815) |
5298035 ns/iter (± 147912) |
10.04 |
This comment was automatically generated by workflow using github-action-benchmark.
…eld from RemoteGraph
…e impls from Py* types to op types
…Degree/OutDegree/Name) with Python bindings + tests
…)/.dst() navigation
…optional-string machinery
…ypes, ExcludeNodes) + Valid, DefaultLayer, and graph Path/Namespace/Name terminals
…ighbour accessors
… wrong answers under view chains
…est, layer, shrink_*, etc.)
…est, layer, shrink_*, etc.)
…p through base_graph on all Remote types
…er parity on path types
…isplay prints the inverse)
…rde as single wire source of truth
…0=auto heuristic, and the op.rs render leak
…per-member deep-clone cliff
…ne blocking_compute per nested read, not per source)
…ollecting the full history
…n InputTime, add filter serde goldens
ljeub-pometry
requested changes
Aug 3, 2026
ljeub-pometry
left a comment
ljeub-pometry
left a comment
Collaborator
There was a problem hiding this comment.
Correctness:
- Make sure property types match between remote/local apis (need to pass along the data type to cast the output correctly)
- Filter apis in graphql need to be aligned with rust/python such that all filters pass through the remote client correctly
- get_dtype_of needs to return PropType, not String
- Change the inner type of
Prop::MaptoIndexMap, so we don't have to worry about scrambling the order everywhere
Performance:
- A lot of String allocations when building the queries can be avoided by passing in a mutable String as a buffer
eventandevent_layeron edges need to be implemented efficiently as part ofEdgeViewOpsinstead of doing a linear search over the exploded edges
Tidy:
- the client modules can be tidied up a bit
- a lot of unnecessary manual conversion to
Py<PyAny>>in the python apis
| local = Graph() | ||
| build(local) | ||
|
|
||
| with GraphServer(tempfile.mkdtemp()).start() as server: |
Collaborator
There was a problem hiding this comment.
use with tempfile.TemporaryDirectory() as work_dir: to make sure your tests actually clean up after themselves
|
|
||
| A context manager — the server is started on enter and torn down on exit. | ||
| """ | ||
| work_dir = tempfile.mkdtemp() |
Collaborator
There was a problem hiding this comment.
use with tempfile.TemporaryDirectory() as work_dir: to make sure your tests actually clean up after themselves
Comment on lines
+35
to
+37
| rg.add_node(1, "ben") | ||
| rg.add_node(2, "hamza") | ||
| rg.add_edge(3, "ben", "hamza") |
Collaborator
There was a problem hiding this comment.
if you take these out of the context manager setup code and put them in a separate function, you can reuse your server startup context manager for all the tests
Collaborator
There was a problem hiding this comment.
might need to pass the graph name and type as input arguments for the context manager
| work_dir = tempfile.mkdtemp() | ||
| with GraphServer(work_dir).start() as server: | ||
| client = server.get_client() | ||
| client.new_graph("g", "EVENT") |
Collaborator
There was a problem hiding this comment.
this should return a remote graph (it might already?)
| } | ||
| "#; | ||
|
|
||
| let variables = HashMap::from([ |
Collaborator
There was a problem hiding this comment.
not your fault, but this seems to cause a lot of intermediate allocations due to the input type for the variables
| // The two view types share an identical method surface and pivot logic; the | ||
| // only difference is which client handle (and thus which container) they wrap. | ||
| // A macro keeps the two `#[pymethods]` blocks in lockstep without duplication. | ||
| macro_rules! columnar_view_methods { |
Collaborator
There was a problem hiding this comment.
this seems to be missing get_dtypes_of
| /// | ||
| /// Returns: | ||
| /// Optional[str]: the property's data-type, or None if absent. | ||
| pub fn get_dtype_of(&self, key: String) -> Result<Option<String>, ClientError> { |
Collaborator
There was a problem hiding this comment.
this needs to return the property type, not a string
Comment on lines
+24
to
+32
| /// Convert a `Prop` value into a native Python object — the raw value a local | ||
| /// `Properties`/`Metadata` `.get()`/`.values()` returns (drop-in parity; no | ||
| /// `RemoteProperty` wrapper). Used by the non-temporal containers. | ||
| fn prop_to_py(py: Python<'_>, value: Prop) -> Result<Py<PyAny>, ClientError> { | ||
| Ok(value | ||
| .into_pyobject(py) | ||
| .map_err(|e| ClientError::InvalidResponse(e.to_string()))? | ||
| .unbind()) | ||
| } |
Collaborator
There was a problem hiding this comment.
this seems completely pointless, can just return the Prop
Collaborator
There was a problem hiding this comment.
although, maybe this should take a dtype and handle casting...
| } | ||
|
|
||
| #[pymethods] | ||
| impl PyRemoteMetadata { |
Collaborator
There was a problem hiding this comment.
all the Py<PyAny>> can just be Prop, let pyo3 generate the conversion code
Comment on lines
+63
to
+70
| pub fn filter(&self, filter: PyFilterExpr) -> PyResult<PyRemoteNestedEdges> { | ||
| let composite = filter | ||
| .try_as_edge_filter() | ||
| .map_err(|e| PyValueError::new_err(e.to_string()))?; | ||
| let gql_filter = composite | ||
| .try_into() | ||
| .map_err(|e: raphtory::errors::GraphError| PyValueError::new_err(e.to_string()))?; | ||
| Ok(PyRemoteNestedEdges::new(self.edges.filter(gql_filter))) |
Collaborator
There was a problem hiding this comment.
need to support all valid filters
…add items() for pairs; drop the RemoteProperty wrapper
…wer: remote_nested_edges.rs:39)
…p: ViewOp }: one vocabulary, data-driven render/parse, ctx inspectable
… Transport contract it implements
…ilding HashMaps per request
…anup; yield RemoteGraph
Co-authored-by: Shivam <4599890+shivamka1@users.noreply.github.com>
… everything — a fail-open for stored access filters
… combinator; regenerate schema
…both conversion directions
…accept string GIDs in id ordering, reject degree op-chains early, normalize Layer::None/All
…rsion; legacy node/edge keys still load via serde aliases
…e/Edge/collections; remove filterNodes/filterEdges; collapse the client's six filter ops into one
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Transport abstraction for the RemoteGraph client
Motivation
The existing Python
RemoteGraphclient is write-only, with each of ~15mutation methods inlining its own Jinja template and JSON-parsing chain.
This PR consolidates that plumbing into a single
Transportabstractionthat every operation flows through, and adds a minimal composable read
surface on top of the same seam so the client can now do lazy view
composition like
rg.window(0, 10).node("ben").degree()end-to-end.What shipped
Transporttrait inraphtory-graphql/src/client/transport.rs—async fn execute(&self, op: &Op) -> Result<Option<Prop>, ClientError>.Op { Read(ReadExpr) | Write(WriteOp) }inop.rs— read side is arecursive expression tree, write side is a flat enum of arg structs.
GraphqlTransportingraphql_transport.rs—renders write ops via the existing Jinja templates (moved from client
wrappers), walks read expressions into nested GraphQL queries, parses
responses.
wire behavior: 7 on
RemoteGraph, 4 onRemoteNode, 4 onRemoteEdge.Each method body shrinks from ~30–50 lines to ~5–15 lines.
Root,Window,Node,Degree— enough forrg.window(s, e).node(id).degree()end-to-end. Only the terminal firesan RPC.
GraphQLRemote{Graph,Node,Edge}→Remote*,RaphtoryGraphQLClient→RemoteClient,raphtory_client.rs→remote_client.rs. Python-facing names unchanged.clientfield removed fromRemoteNode/RemoteEdge— theynow hold only
transportand view state. Kept onRemoteGraphpendingbatch-method migration (see below).
PyRemoteGraph::node/edgenow delegate to the Rust methods,so the accumulated view chain (
window(...)) is correctly propagatedwhen descending to a node — previously silently dropped.
window()anddegree().Testing
raphtory-graphqltests still green.graphql_transport.rs(unit tests for the readrender/parse; one integration test spawning a real server).
test_remote_graph_transport.py:test_add_and_degree,test_windowed_degree,test_view_chain_propagation.TODO
add_nodes,add_edges) still callclient.query(...)directly (existingTODOs). Migrating them removesthe last
clientfield.properties, ...).
return-type conversion pattern.
for n in g.nodes: ....