From 200df9e70503c03e56a670b78befff100e1bbb72 Mon Sep 17 00:00:00 2001 From: zelig Date: Fri, 18 Apr 2025 06:48:48 +0200 Subject: [PATCH 01/23] create --- SWIPs/swip-39.md | 54 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 SWIPs/swip-39.md diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md new file mode 100644 index 00000000..89b30dc8 --- /dev/null +++ b/SWIPs/swip-39.md @@ -0,0 +1,54 @@ +--- +SWIP: +title: +author: , FirstName (@GitHubUsername) and GitHubUsername (@GitHubUsername)> +discussions-to: +status: Draft +type: +category (*only required for Standard Track): +created: +requires (*optional): +replaces (*optional): +--- + + +This is the suggested template for new SWIPs. + +Note that a SWIP number will be assigned by an editor. When opening a pull request to submit your SWIP, please use an abbreviated title in the filename, `SWIP-draft_title_abbrev.md`. + +The title should be 44 characters or less. + +## Simple Summary + +If you can't explain it simply, you don't understand it well enough." Provide a simplified and layman-accessible explanation of the SWIP. + +## Abstract + +A short (~200 word) description of the technical issue being addressed. + +## Motivation + +The motivation is critical for SWIPs that want to change the Swarm protocol. It should clearly explain why the existing protocol specification is inadequate to address the problem that the SWIP solves. SWIP submissions without sufficient motivation may be rejected outright. + +## Specification + +The technical specification should describe the syntax and semantics of any new feature. The specification should be detailed enough to allow competing, interoperable implementations for the current Swarm platform and future client implementations. + +## Rationale + +The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale may also provide evidence of consensus within the community, and should discuss important objections or concerns raised during discussion. + +## Backwards Compatibility + +All SWIPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The SWIP must explain how the author proposes to deal with these incompatibilities. SWIP submissions without a sufficient backwards compatibility treatise may be rejected outright. + +## Test Cases + +Test cases for an implementation are mandatory for SWIPs that are affecting changes to data and message formats. Other SWIPs can choose to include links to test cases if applicable. + +## Implementation + +The implementations must be completed before any SWIP is given status "Final", but it need not be completed before the SWIP is accepted. While there is merit to the approach of reaching consensus on the specification and rationale before writing code, the principle of "rough consensus and running code" is still useful when it comes to resolving many discussions of API details. + +## Copyright +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From c2bf1acf42827eef8b2591ed201208041d6eeca2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Mon, 21 Jul 2025 09:53:02 +0200 Subject: [PATCH 02/23] Update swip-39.md --- SWIPs/swip-39.md | 440 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 402 insertions(+), 38 deletions(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index 89b30dc8..cb517688 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -1,54 +1,418 @@ --- -SWIP: -title: -author: , FirstName (@GitHubUsername) and GitHubUsername (@GitHubUsername)> -discussions-to: +SWIP: 39 +title: Balanced Neighbourhood Registry aka Smart Neighbourhood Management +author: Viktor Trón (@zelig) +discussions-to: https://discord.gg/Q6BvSkCv status: Draft -type: -category (*only required for Standard Track): -created: -requires (*optional): -replaces (*optional): +type: Standards Track +category: Core +created: 2025-07-21 --- - -This is the suggested template for new SWIPs. +# Balanced Neighbourhood Registry aka Smart Neighbourhood Management -Note that a SWIP number will be assigned by an editor. When opening a pull request to submit your SWIP, please use an abbreviated title in the filename, `SWIP-draft_title_abbrev.md`. +## Abstract -The title should be 44 characters or less. +This SWIP introduces a systematic way for operators to enter a service network in such a way that their overlay addresses are balanced within the address space. Importantly, operators need to register their commitment to take part and are assigned a random neighbourhood (of depth $d=int(log_2(N))+1$). -## Simple Summary - -If you can't explain it simply, you don't understand it well enough." Provide a simplified and layman-accessible explanation of the SWIP. +## Motivation -## Abstract - -A short (~200 word) description of the technical issue being addressed. +There are multiple considerations that motivate such a scheme: +- **load-balancing**: any decentralised service network will be fair if tasks are distributed to nodes so that the workload assigned to each participant is roughly equal. Such load balancing is achieved if tasks are assigned based on uniform random label (i.e., the content hash of the descriptor)[^1] and nodes providing the service are balanced in the address space. -## Motivation - -The motivation is critical for SWIPs that want to change the Swarm protocol. It should clearly explain why the existing protocol specification is inadequate to address the problem that the SWIP solves. SWIP submissions without sufficient motivation may be rejected outright. +[^1]: and the average size (computation/storage requirement) of tasks over a typical period of payment has tolerable variance. + +- **arbitrary neighbourhood assigment**: the system needs to make sure that assignment of an overlay address to participant nodes is arbitrary. In particular, it is impractical (expensive) for any operator to attempt to place several nodes in the same storage neighbourhood (neighbourhood of depth $d=int(log_2(N))+1$). Note that, this scheme constitutes an effective solution to the problem of "one operator, one node in a neighbourhood". Taking the storage incentive system as an example, this will take care of the sybil issue[^2] without resorting to the rather weak incentive of additive stake as a proof of redundancy.[^3] + +[^2]: that operators run several nodes in one neighbourhood without truely replicating storage and yet get paid. + +[^3]: the idea is that if stake is variable and earnings are linearly proportional to earnings then, mutatis mutandis, it is always more economical for one operator to run just one node with all the stake than several nodes due to the added operational costs. + +## Architecture + +The balanced neighbourhood assignment (associated with a service) is orchestrated by a smart contract which is deployed together with a [staking contract](https://github.com/ethersphere/storage-incentives/blob/master/src/Staking.sol). The contract API provides 2 transactional endpoints relevant for entry: + +One registers a node by its ether address ($a_\xi$) in the *committers' list* $C$ that records nodes' commitment to partpicipate as a provider in the associated decentralised service network. + +The other one is called by the staking contract after the service network stake is deposited with a valid nonce, i.e., one that will put the node in the right neighbourhood. This call will place the node among the active node set for the service, and removes the entry from the committers list. This function includes a read-only call that takes as argument a node's ether address and returns a neighbourhood the node is assigned to. This call is public so that the client can enquire about the neighbourhood they are assigned to -- so that they can mine an overlay address into it, ie., find a nonce that is needed to generate the overlay address. ## Specification - -The technical specification should describe the syntax and semantics of any new feature. The specification should be detailed enough to allow competing, interoperable implementations for the current Swarm platform and future client implementations. -## Rationale - -The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale may also provide evidence of consensus within the community, and should discuss important objections or concerns raised during discussion. +### Registration +An initially empty list (*committers' list*) of *entry struct* types holds the current committers. The struct holds information about the ether address of the node, the blockheight the address registered at. + + + +#### Deposit +In order for a node to get its address registered, an amount of $S_0$ (stake zero) must be deposited. + +#### Uniqueness +In order to prevent repeated trials, each node must be registered only once. + +After checking the deposit amount and the uniqueness check on the ether address, the current blockheight is recorded with the address by pushing the entry struct ($e_n=$) to the end of committers list. + +#### Validity +The entry is valid for a period of $B$ blocks after the registration[^55]. + +[^55]: $B$ is less than $256$, the number of blocks available from within the EVM). + +Since the blockheight values of the list items are monotonically increasing, entries at the beginning of the list expire first. By iterating upto the first valid entry, expired entries can be iterated on efficiently. + +### Expiry + +This function call iterates through all expired entries, burns their deposit, and, by setting the head of the list to the first valid item, removes them from the committer's list. + +This is called by the assign function (itself called by the staking contract) before the read only call checking if the resulting overlay address falls into the neighbourhood that the registrant was assigned to, i.e., the correctness of the nonce submitted from the perspective of the staking contract. + +### Assign + +The assign call is the second transactional endpoint called by the staking contract. It takes the provider's ether address and as well as the mined overlay as arguments. +After calling expiry, the validity of the registration is checked by finding the entry for the ether address in the committers' list. + +#### Initialisation + +Let $\overline{N}=2^d$ be the lowest power of 2 that is greater than $N$, the number of already assigned registrants, and let $d$ be its exponent ($d=int(log_2(N))+1$). Let $R$ be the array of the remainding unassigned neighbouprhoods of this level (i.e., $len(R)=\overline{N}-N$). Whenever $len(R)$ drops to $0$, $d$ is incremented and $\overline{N}=2^d$ adjusts. At any point in time, a $uint256$ array of length $\overline{N}$ is maintained, called the *assignments list* $A$ holding the registered node's overlay addresses. +Whenever $d$ changes, new arrays for assigments $A_d$ and remainders $R_d$ are created (both twice the size of the previous one $A_{d-1}$ and $R_{d-1}$). We iterate through the current array and for each overlay address $a^O_i$ at position $i$ copy $a^O_i$ to position $2\cdot i +b_i$ where $b_i=1$ if the $d$-th bit of the address is set: + +$$ +b_i=a^O_i[d/8] +$$ + +$$ +b_i\gg=7-(d\mod 8) +$$ + +$$ +b_i\,\&\hspace{-3pt}=\mathtt{0x1} +$$ + +In fact, $A_d$ stands for nodes of a binary tree on level $d$ and the value at each position is the overlay address filling that neighbourhood. When $d$ is incremented, we will need to fill some of the neighbourhoods with the existing nodes. Each node will fill the left child node (when $b_i=0$) or the right one ($b_i=1) depending on the subsequent bit in their overlay. + +$$ +\forall 0\leq i < 2^d, A_d[2\cdot i+b_i]=A_{d-1}[i] +$$ + +As we are filling the assignment list, we know that the whenever a neighbourhood is filled with an already existing node, its sister neighbourhood will be unassigned, therefore we can just record those in the remaining list. + +$$ +\forall 0\leq i < R_d[i]=2\cdot i+1-b_i +$$ + + +#### Random seed + +This internal read-only call takes as its single argument an ether address (a_\Xi) and returns a random $uint256$. +After checking if the ether address is a valid registrant by finding the corresponding first (unique) $entry$ struct,[^33] the `difficulty` (randao) is called on the block subsequent to the blockheight registered. Append to it the ether address and hash it to yield what will serve as the random seed for this provider.[^4] + +$$ +\sigma:=\mathit{blockAtHeight}(\mathit{entry}(a_\Xi).\mathit{height}+1).\mathit{difficulty}() +$$ + +$$ +\varrho:=\mathit{uint256}(H(\sigma|a_\Xi)) +$$ + +[^33]: Since expiry is not necessarily called when the random seed it called, the blockheight needs to be checked: 1) if block after $h$ (the one in which the registration happened) is available, 2) that the height is not greater than $h+D$ with $D$ being the validity period in blocks. + +[^4]: Even if on a POA chain, and no randao, this seed cannot be known to the registering provider and their colluding associates, but nonetheless should be deterministic once its set. + +#### Neighbourhood + +This component must be available as a public readonly endpoint taking a node's ($n$) ether address ($a^\Xi_n$) as a single argument. +The random nonce $\varrho_n$ is used to select a neighbourhood $nh$ for a provider from the remaining unassigned neighbourhood list of level $d$: + +$$ +i:=\varrho\mod len(R_d) +$$ + +$$ +nh=R\,[\,i\,] +$$ + +#### Checking the overlay + +The overlay (obtained by mining the nonce) is checked to fall in the correct neighbourhood r: +The check validates the address $a^O_n$ if and only if: + +$$ +r=a^O_n\gg(255-d) +$$ + +#### Assignment + +If the overlay check passes, + +- the nodes' overlay address is assigned to a neighbourhood of depth $d$. + +$$ +A_d[r]=a^O_n +$$ + +- $N$ is incremented +- the $i$-th item is removed from remaining open neighbourhood list $R_d$. +- if $R_d$ is now of zero length, the $d$ is incremented, and new assignment and remaining lists are initialised as per section 'initialisation' above. +- provider's entry is removed from the committers' list. + + +### Further endpoints + +A public read-only endpoint exists for querying neighbourhoods as well as nodes. Accessor for $d$ and $N$ will return the current neighbourhood depth and the current number of assigned neighbourhoods. A public accessor for $A_d$ will return for a neighbourhood (between $0$ and $2^d-1 inclusive) the overlay of the node assigned to that neighbourhood. Another endpoint will return for any overlay $o$ the closest node, so that the network service can find responsible nodes for any task with address in the space shared by overlays: + +$$ +g(a)=A_d[a\gg(255-d)] +$$ + +### Deregistration + +Only called from the Staking contract, deregister deletes the entry for the neighbourhood belonging to the given address, makes the neighbourhood available in $R$ + + +## Implementation notes + +### Changes to the staking contract + +### Changes to the bee client + +A new endpoint to bee client must be added to register a node that is not yet registered to be assigned a neighbourhood. Once the neighbourhood is known, the client can mine the nonce needed to place the overlay in the required neighbourhood. + +### Migration + +Since a new updated staking contract, a stake migration will be needed for the upgrade. Before the change, all the simplification of the staking contract is recommended, especially to allow fixed stake in order to realign redundancy +of storage and monetary incentive: with a fixed amount staked, total stake is linearly proportional to the number of nodes, and therefore comparisons across neighbourhoods can be made based on the number of nodes. In particular, the arbitrary balanced assignment makes sense in terms of incentives (expected revenue). + +### Putting a node in each neighbourhood. + +## Contract + +```sol +// SPDX-License-Identifier: MIT +pragma solidity ^0.8.19; + +contract BalancedNeighbourhoodRegistry { + // -------------------- + // Configurable values + // -------------------- + uint256 public constant STAKE = 10,000,000,000,000,000 wei; // Stake required + uint256 public constant VALID_FOR = 128; // Validity window in blocks + + // -------------------- + // Structs and Storage + // -------------------- + struct Entry { + address committer; + uint256 height; + } + + Entry[] public committers; // List of committers (registered but as yet unassigned nodes) + mapping(address => bool) public hasCommitted; // Track if an address has committed + + uint256 public N; // Number of assigned nodes + uint256 public d = 1; // Neighbourhood depth + uint256 public currentPower = 2; // 2^d + + // Assignments: overlay address for neighbourhoods of depth d + mapping(uint256 => address) public A; + + // Remaining unassigned neighbourhoods of depth d + uint256[] public R; + + // Overlay lookup + mapping(address => bytes32) public overlays; + + // -------------------- + // Events + // -------------------- + event Registered(address indexed node, uint256 blockHeight); + event Assigned(address indexed node, uint256 neighbourhood, bytes32 overlay); + event DepthUpgraded(uint256 newDepth); + event Deregistered(address indexed node, uint256 neighbourhood); + + // -------------------- + // Registration endpoint + // -------------------- + function register() external payable { + require(msg.value == STAKE, "Invalid stake"); + require(!hasCommitted[msg.sender], "Already registered"); + + committers.push(Entry(msg.sender, block.number)); + hasCommitted[msg.sender] = true; + + emit Registered(msg.sender, block.number); + } + + // -------------------- + // Expire old entries + // -------------------- + function _expire() internal { + while (committers.length > 0) { + Entry storage e = committers[0]; + if (block.number <= e.height + VALID_FOR) { + break; + } + hasCommitted[e.committer] = false; + _removeCommitter(0); + // Burn logic: funds stay locked. + } + } + + // -------------------- + // Remove committer from the list + // -------------------- + // This function is used internally to remove a committer from the list. + // It shifts the elements to the left and pops the last element. + function _removeCommitter(uint index) internal { + if (index >= committers.length) return; + + for (uint i = index; i < committers.length - 1; i++) { + committers[i] = committers[i + 1]; + } + committers.pop(); + } + + // -------------------- + // Find entry for a committer in the committer list + // Returns the index of the entry if found, otherwise reverts. + // -------------------- + function _findEntryFor(address _a) internal view returns (uint) { + for (uint i = 0; i < committers.length; i++) { + if (committers[i].committer == _a) { + return ì; + } + } + revert("Not registered"); + } + + // -------------------- + // Randomness + // -------------------- + function _randomSeed(address _a) internal view returns (uint256) { + uint memory i = _findEntryFor(_a); + uint256 h = committers[i].height; + // Ensure the block number is valid as expire may not have been called + require(block.number > h + 1, "Too early"); + require(block.number <= h + VALID_FOR, "Registration expired"); + // Use blockhash to generate a random seed + bytes32 bh = blockhash(h + 1); + return uint256(keccak256(abi.encodePacked(bh, _a))); + } + + // -------------------- + // Public View: Neighbourhood + // -------------------- + function getNeighbourhood(address _a) public view returns (uint256) { + uint256 r = _randomSeed(_a); + require(R.length > 0, "No available neighbourhoods"); + return R[r % R.length]; + } + + // -------------------- + // Assign node overlay to neighbourhood + // -------------------- + function assign(address _a, bytes32 _overlay) external { + _expire(); + + // Check registration + uint256 nh = getNeighbourhood(_a); + uint256 overlayNh = uint256(_overlay) >> (256 - d); + + require(overlayNh == nh, "Overlay doesn't match neighbourhood"); + + A[nh] = _overlay; + overlays[_a] = _overlay; + N++; + + // Remove neighbourhood from R + _removeFromR(nh); + + // Check if R is empty + if (R.length == 0) { + _upgradeDepth(); + } + + hasCommitted[_a] = false; + _removeEntry(_a); + + emit Assigned(_a, nh, _overlay); + } + + function deregister(address a) external { + overlay = overlays[a]; + require(overlay != bytes32(0), "Not registered"); + overlays[a] = bytes32(0); + uint nh = uint256(overlay) >> (256 - d); + A[nh] = bytes32(0); + R.push(nh); + N--; + // return funds to the committer + payable(a).transfer(STAKE); + emit Deregistered(a, nh); + } + + // -------------------- + // Internal functions to manage R and committers + // -------------------- + function _removeFromR(uint256 nh) internal { + for (uint i = 0; i < R.length; i++) { + if (R[i] == nh) { + // replace the removed element with the last element and pop + R[i] = R[R.length - 1]; + R.pop(); + return; + } + } + } + + function _removeEntry(address _a) internal { + uint i = _findEntryFor(_a); + require(i < committers.length, "Entry not found"); + _removeCommitter(i); + } + + // -------------------- + // Expand A and R when needed + // -------------------- + function _upgradeDepth() internal { + currentPower = 2 ** d; + + R = new uint[](currentPower); + // A.length *= 2 // does this not work to extend to its double with zero values? + for (uint i = 0; i < currentPower; i++) { + // Ensure A has enough space for the new neighbourhoods); + A.push(bytes32(0)); + } + for (uint i = currentPower - 1; i < currentPower; i--) { + uint b = uint256(A[i]) >> (256 - d) % 2; + A[2*i+b] = A[i]; + j = 2*i+1-b; + if (2 * j < currentPower) { + A[j] = bytes32(0); // Clear the old address + } + R[i] = j + } + d++; + require(d <= 32, "Maximum depth exceeded"); + emit DepthUpgraded(d); + } + + // -------------------- + // Accessors + // -------------------- + function getDepth() external view returns (uint) { + return d; + } -## Backwards Compatibility - -All SWIPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The SWIP must explain how the author proposes to deal with these incompatibilities. SWIP submissions without a sufficient backwards compatibility treatise may be rejected outright. + function getN() external view returns (uint256) { + return N; + } -## Test Cases - -Test cases for an implementation are mandatory for SWIPs that are affecting changes to data and message formats. Other SWIPs can choose to include links to test cases if applicable. + function getOverlayForNeighbourhood(uint256 nh) external view returns (bytes32) { + return A[nh]; + } -## Implementation - -The implementations must be completed before any SWIP is given status "Final", but it need not be completed before the SWIP is accepted. While there is merit to the approach of reaching consensus on the specification and rationale before writing code, the principle of "rough consensus and running code" is still useful when it comes to resolving many discussions of API details. + function getClosestNode(bytes32 key) external view returns (bytes32) { + uint256 prefix = uint256(key) >> (256 - d); + return A[prefix]; + } +} +``` -## Copyright -Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From 7cca5af21453781f33847b5992f53b0f2cdfd6e7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Mon, 21 Jul 2025 20:44:21 +0200 Subject: [PATCH 03/23] Update SWIPs/swip-39.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- SWIPs/swip-39.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index cb517688..dae8395e 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -32,7 +32,7 @@ There are multiple considerations that motivate such a scheme: The balanced neighbourhood assignment (associated with a service) is orchestrated by a smart contract which is deployed together with a [staking contract](https://github.com/ethersphere/storage-incentives/blob/master/src/Staking.sol). The contract API provides 2 transactional endpoints relevant for entry: -One registers a node by its ether address ($a_\xi$) in the *committers' list* $C$ that records nodes' commitment to partpicipate as a provider in the associated decentralised service network. +One registers a node by its ether address ($a_\xi$) in the *committers' list* $C$ that records nodes' commitment to participate as a provider in the associated decentralised service network. The other one is called by the staking contract after the service network stake is deposited with a valid nonce, i.e., one that will put the node in the right neighbourhood. This call will place the node among the active node set for the service, and removes the entry from the committers list. This function includes a read-only call that takes as argument a node's ether address and returns a neighbourhood the node is assigned to. This call is public so that the client can enquire about the neighbourhood they are assigned to -- so that they can mine an overlay address into it, ie., find a nonce that is needed to generate the overlay address. From 955db1561d1ed0791928d2da4e544f4a487f5d85 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Mon, 21 Jul 2025 20:44:44 +0200 Subject: [PATCH 04/23] Update SWIPs/swip-39.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- SWIPs/swip-39.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index dae8395e..57a4dd33 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -71,7 +71,7 @@ After calling expiry, the validity of the registration is checked by finding the #### Initialisation -Let $\overline{N}=2^d$ be the lowest power of 2 that is greater than $N$, the number of already assigned registrants, and let $d$ be its exponent ($d=int(log_2(N))+1$). Let $R$ be the array of the remainding unassigned neighbouprhoods of this level (i.e., $len(R)=\overline{N}-N$). Whenever $len(R)$ drops to $0$, $d$ is incremented and $\overline{N}=2^d$ adjusts. At any point in time, a $uint256$ array of length $\overline{N}$ is maintained, called the *assignments list* $A$ holding the registered node's overlay addresses. +Let $\overline{N}=2^d$ be the lowest power of 2 that is greater than $N$, the number of already assigned registrants, and let $d$ be its exponent ($d=int(log_2(N))+1$). Let $R$ be the array of the remaining unassigned neighbourhoods of this level (i.e., $len(R)=\overline{N}-N$). Whenever $len(R)$ drops to $0$, $d$ is incremented and $\overline{N}=2^d$ adjusts. At any point in time, a $uint256$ array of length $\overline{N}$ is maintained, called the *assignments list* $A$ holding the registered node's overlay addresses. Whenever $d$ changes, new arrays for assigments $A_d$ and remainders $R_d$ are created (both twice the size of the previous one $A_{d-1}$ and $R_{d-1}$). We iterate through the current array and for each overlay address $a^O_i$ at position $i$ copy $a^O_i$ to position $2\cdot i +b_i$ where $b_i=1$ if the $d$-th bit of the address is set: $$ From d4cd473f9154f99a181fd92638e35e22941df811 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Mon, 21 Jul 2025 20:45:03 +0200 Subject: [PATCH 05/23] Update SWIPs/swip-39.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- SWIPs/swip-39.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index 57a4dd33..2c4a0dbf 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -276,7 +276,7 @@ contract BalancedNeighbourhoodRegistry { function _findEntryFor(address _a) internal view returns (uint) { for (uint i = 0; i < committers.length; i++) { if (committers[i].committer == _a) { - return ì; + return i; } } revert("Not registered"); From 537a60fbf52166c1c00d94eb1f39a4f37d5318a8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Mon, 21 Jul 2025 20:45:40 +0200 Subject: [PATCH 06/23] Update SWIPs/swip-39.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- SWIPs/swip-39.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index 2c4a0dbf..77a37da9 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -286,7 +286,7 @@ contract BalancedNeighbourhoodRegistry { // Randomness // -------------------- function _randomSeed(address _a) internal view returns (uint256) { - uint memory i = _findEntryFor(_a); + uint i = _findEntryFor(_a); uint256 h = committers[i].height; // Ensure the block number is valid as expire may not have been called require(block.number > h + 1, "Too early"); From 5781c8b1b5e67f4a1e41ed94da641fcccf2dd732 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Mon, 21 Jul 2025 20:47:08 +0200 Subject: [PATCH 07/23] Update SWIPs/swip-39.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- SWIPs/swip-39.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index 77a37da9..b7e6c0d1 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -383,7 +383,7 @@ contract BalancedNeighbourhoodRegistry { for (uint i = currentPower - 1; i < currentPower; i--) { uint b = uint256(A[i]) >> (256 - d) % 2; A[2*i+b] = A[i]; - j = 2*i+1-b; + uint256 j = 2*i+1-b; if (2 * j < currentPower) { A[j] = bytes32(0); // Clear the old address } From a71cad1dfc36b16155aa74c7ded551cbe7d26295 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Mon, 21 Jul 2025 20:47:24 +0200 Subject: [PATCH 08/23] Update SWIPs/swip-39.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- SWIPs/swip-39.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index b7e6c0d1..3022bb0d 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -387,7 +387,7 @@ contract BalancedNeighbourhoodRegistry { if (2 * j < currentPower) { A[j] = bytes32(0); // Clear the old address } - R[i] = j + R[i] = j; } d++; require(d <= 32, "Maximum depth exceeded"); From fff7410af0d6e0235bb812c7ae4f42a35f5b0933 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Mon, 21 Jul 2025 20:50:29 +0200 Subject: [PATCH 09/23] Update SWIPs/swip-39.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- SWIPs/swip-39.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index 3022bb0d..73903d5a 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -211,7 +211,7 @@ contract BalancedNeighbourhoodRegistry { uint256 public currentPower = 2; // 2^d // Assignments: overlay address for neighbourhoods of depth d - mapping(uint256 => address) public A; + mapping(uint256 => bytes32) public A; // Remaining unassigned neighbourhoods of depth d uint256[] public R; From c0edaffea28a96ddab0ecc2e940f70b96965b5ca Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Mon, 21 Jul 2025 22:38:59 +0200 Subject: [PATCH 10/23] Update swip-39.md --- SWIPs/swip-39.md | 43 ++++++++++++++++++++----------------------- 1 file changed, 20 insertions(+), 23 deletions(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index 73903d5a..f4316e82 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -32,7 +32,7 @@ There are multiple considerations that motivate such a scheme: The balanced neighbourhood assignment (associated with a service) is orchestrated by a smart contract which is deployed together with a [staking contract](https://github.com/ethersphere/storage-incentives/blob/master/src/Staking.sol). The contract API provides 2 transactional endpoints relevant for entry: -One registers a node by its ether address ($a_\xi$) in the *committers' list* $C$ that records nodes' commitment to participate as a provider in the associated decentralised service network. +One registers a node by its ether address ($a^\Xi$) in the *committers' list* $C$ that records nodes' commitment to participate as a provider in the associated decentralised service network. The other one is called by the staking contract after the service network stake is deposited with a valid nonce, i.e., one that will put the node in the right neighbourhood. This call will place the node among the active node set for the service, and removes the entry from the committers list. This function includes a read-only call that takes as argument a node's ether address and returns a neighbourhood the node is assigned to. This call is public so that the client can enquire about the neighbourhood they are assigned to -- so that they can mine an overlay address into it, ie., find a nonce that is needed to generate the overlay address. @@ -49,7 +49,7 @@ In order for a node to get its address registered, an amount of $S_0$ (stake ze #### Uniqueness In order to prevent repeated trials, each node must be registered only once. -After checking the deposit amount and the uniqueness check on the ether address, the current blockheight is recorded with the address by pushing the entry struct ($e_n=$) to the end of committers list. +After checking the deposit amount and the uniqueness check on the ether address, the current blockheight is recorded with the address by pushing the entry struct ($e_n=$) to the end of committers list. #### Validity The entry is valid for a period of $B$ blocks after the registration[^55]. @@ -71,7 +71,7 @@ After calling expiry, the validity of the registration is checked by finding the #### Initialisation -Let $\overline{N}=2^d$ be the lowest power of 2 that is greater than $N$, the number of already assigned registrants, and let $d$ be its exponent ($d=int(log_2(N))+1$). Let $R$ be the array of the remaining unassigned neighbourhoods of this level (i.e., $len(R)=\overline{N}-N$). Whenever $len(R)$ drops to $0$, $d$ is incremented and $\overline{N}=2^d$ adjusts. At any point in time, a $uint256$ array of length $\overline{N}$ is maintained, called the *assignments list* $A$ holding the registered node's overlay addresses. +Let $\overline{N}=2^d$ be the lowest power of 2 that is greater than $N$, the number of already assigned registrants, and let $d$ be its exponent ($d=int(log_2(N))+1$). Let $R$ be the array of the remaining unassigned neighbourhoods of this level (i.e., $len(R)=\overline{N}-N$). Whenever $len(R)$ drops to $0$, $d$ is incremented and $\overline{N}=2^d$ adjusts. At any point in time, a $uint256$ array of length $\overline{N}$ is maintained, called the *assignments list* $A$ holding the registered nodes' overlay addresses. Whenever $d$ changes, new arrays for assigments $A_d$ and remainders $R_d$ are created (both twice the size of the previous one $A_{d-1}$ and $R_{d-1}$). We iterate through the current array and for each overlay address $a^O_i$ at position $i$ copy $a^O_i$ to position $2\cdot i +b_i$ where $b_i=1$ if the $d$-th bit of the address is set: $$ @@ -82,11 +82,12 @@ $$ b_i\gg=7-(d\mod 8) $$ + $$ -b_i\,\&\hspace{-3pt}=\mathtt{0x1} +b_i{\land\hspace{-3pt}=}1 $$ -In fact, $A_d$ stands for nodes of a binary tree on level $d$ and the value at each position is the overlay address filling that neighbourhood. When $d$ is incremented, we will need to fill some of the neighbourhoods with the existing nodes. Each node will fill the left child node (when $b_i=0$) or the right one ($b_i=1) depending on the subsequent bit in their overlay. +In fact, $A_d$ stands for nodes of a binary tree on level $d$ and the value at each position is the overlay address filling that neighbourhood. When $d$ is incremented, we will need to fill some of the neighbourhoods with the existing nodes. Each node will fill the left child node (when $b_i=0$) or the right one ($b_i=1$) depending on the subsequent bit in their overlay. $$ \forall 0\leq i < 2^d, A_d[2\cdot i+b_i]=A_{d-1}[i] @@ -95,21 +96,21 @@ $$ As we are filling the assignment list, we know that the whenever a neighbourhood is filled with an already existing node, its sister neighbourhood will be unassigned, therefore we can just record those in the remaining list. $$ -\forall 0\leq i < R_d[i]=2\cdot i+1-b_i +\forall 0\leq i < len(R_d), R_d[i]=2\cdot i+1-b_i $$ #### Random seed -This internal read-only call takes as its single argument an ether address (a_\Xi) and returns a random $uint256$. +This internal read-only call takes as its single argument an ether address ($a^\Xi$) and returns a random $uint256$. After checking if the ether address is a valid registrant by finding the corresponding first (unique) $entry$ struct,[^33] the `difficulty` (randao) is called on the block subsequent to the blockheight registered. Append to it the ether address and hash it to yield what will serve as the random seed for this provider.[^4] $$ -\sigma:=\mathit{blockAtHeight}(\mathit{entry}(a_\Xi).\mathit{height}+1).\mathit{difficulty}() +\sigma:=\mathit{blockAtHeight}(\mathit{entry}(a^\Xi).\mathit{height}+1).\mathit{difficulty}() $$ $$ -\varrho:=\mathit{uint256}(H(\sigma|a_\Xi)) +\varrho:=\mathit{uint256}(H(\sigma|a^\Xi)) $$ [^33]: Since expiry is not necessarily called when the random seed it called, the blockheight needs to be checked: 1) if block after $h$ (the one in which the registration happened) is available, 2) that the height is not greater than $h+D$ with $D$ being the validity period in blocks. @@ -126,7 +127,7 @@ i:=\varrho\mod len(R_d) $$ $$ -nh=R\,[\,i\,] +nh=R[i] $$ #### Checking the overlay @@ -211,21 +212,18 @@ contract BalancedNeighbourhoodRegistry { uint256 public currentPower = 2; // 2^d // Assignments: overlay address for neighbourhoods of depth d - mapping(uint256 => bytes32) public A; + bytes32[] public A; // Remaining unassigned neighbourhoods of depth d uint256[] public R; - // Overlay lookup - mapping(address => bytes32) public overlays; - // -------------------- // Events // -------------------- event Registered(address indexed node, uint256 blockHeight); event Assigned(address indexed node, uint256 neighbourhood, bytes32 overlay); event DepthUpgraded(uint256 newDepth); - event Deregistered(address indexed node, uint256 neighbourhood); + event Unassigned(address indexed node, uint256 neighbourhood); // -------------------- // Registration endpoint @@ -318,7 +316,6 @@ contract BalancedNeighbourhoodRegistry { require(overlayNh == nh, "Overlay doesn't match neighbourhood"); A[nh] = _overlay; - overlays[_a] = _overlay; N++; // Remove neighbourhood from R @@ -335,17 +332,17 @@ contract BalancedNeighbourhoodRegistry { emit Assigned(_a, nh, _overlay); } - function deregister(address a) external { - overlay = overlays[a]; - require(overlay != bytes32(0), "Not registered"); - overlays[a] = bytes32(0); - uint nh = uint256(overlay) >> (256 - d); + //---------------------- + // unregister + //---------------------- + function unregister(address _a, bytes32 _overlay) external { + uint nh = uint256(_overlay) >> (256 - d); A[nh] = bytes32(0); R.push(nh); N--; // return funds to the committer - payable(a).transfer(STAKE); - emit Deregistered(a, nh); + payable(_a).transfer(STAKE); + emit Unassigned(_a, nh); } // -------------------- From f97b73d7644718614c529b3c9a8c6d2d74e078ff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Fri, 25 Jul 2025 01:36:52 +0200 Subject: [PATCH 11/23] Update SWIPs/swip-39.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- SWIPs/swip-39.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index f4316e82..c7fdb28c 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -371,9 +371,9 @@ contract BalancedNeighbourhoodRegistry { function _upgradeDepth() internal { currentPower = 2 ** d; - R = new uint[](currentPower); - // A.length *= 2 // does this not work to extend to its double with zero values? + delete R; for (uint i = 0; i < currentPower; i++) { + R.push(0); // Ensure A has enough space for the new neighbourhoods); A.push(bytes32(0)); } From 77f3efcf1af50b1d193cb216c761f5fe4e0a4d9e Mon Sep 17 00:00:00 2001 From: significance Date: Mon, 11 Aug 2025 11:46:23 +0100 Subject: [PATCH 12/23] trivial changes, standardised some orthography, mathematical expression (#75) --- SWIPs/swip-39.md | 91 ++++++++++++++++++++++++++++++++++-------------- 1 file changed, 64 insertions(+), 27 deletions(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index c7fdb28c..d48749f1 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -13,28 +13,55 @@ created: 2025-07-21 ## Abstract -This SWIP introduces a systematic way for operators to enter a service network in such a way that their overlay addresses are balanced within the address space. Importantly, operators need to register their commitment to take part and are assigned a random neighbourhood (of depth $d=int(log_2(N))+1$). +This SWIP introduces a systematic way for operators to enter the network in such a way that their overlay addresses are balanced within the address space. Importantly, operators must register their commitment to take part and on doing so will be assigned a random neighbourhood from the queue of those next to be occupied. Since a joining node operator is unable to position themselves at a specific point in the network without a significant time or financial penalty, attack vectors that rely on this are rendered infeasible. ## Motivation There are multiple considerations that motivate such a scheme: -- **load-balancing**: any decentralised service network will be fair if tasks are distributed to nodes so that the workload assigned to each participant is roughly equal. Such load balancing is achieved if tasks are assigned based on uniform random label (i.e., the content hash of the descriptor)[^1] and nodes providing the service are balanced in the address space. -[^1]: and the average size (computation/storage requirement) of tasks over a typical period of payment has tolerable variance. +- **load-balancing**: a decentralised service network will be fair if tasks are distributed to nodes so that the workload assigned to each participant is roughly equal. Presuming load balancing is achieved if tasks[^1] are assigned based on uniform random label (i.e., the content hash of the descriptor) and nodes providing the service are balanced in the address space. -- **arbitrary neighbourhood assigment**: the system needs to make sure that assignment of an overlay address to participant nodes is arbitrary. In particular, it is impractical (expensive) for any operator to attempt to place several nodes in the same storage neighbourhood (neighbourhood of depth $d=int(log_2(N))+1$). Note that, this scheme constitutes an effective solution to the problem of "one operator, one node in a neighbourhood". Taking the storage incentive system as an example, this will take care of the sybil issue[^2] without resorting to the rather weak incentive of additive stake as a proof of redundancy.[^3] +[^1]: it is assumed that average resource utilisation (network/computation/storage requirements) of tasks over a typical period of payment remains within a tolerable variance. -[^2]: that operators run several nodes in one neighbourhood without truely replicating storage and yet get paid. +- **arbitrary neighbourhood assigment**: the system needs to make sure that assignment of an overlay address to participant nodes is arbitrary. In particular, it is impractical (expensive) for any operator to attempt to place several nodes in the same storage neighbourhood without truely replicating storage and yet get paid. Note that, this scheme constitutes an effective solution to the problem of "one operator, one node in a neighbourhood". Taking the storage incentive system as an example, this will mitigate this sybil attack, without resorting to the rather weak incentive of additive stake as a proof of redundancy.[^2] -[^3]: the idea is that if stake is variable and earnings are linearly proportional to earnings then, mutatis mutandis, it is always more economical for one operator to run just one node with all the stake than several nodes due to the added operational costs. +[^2]: the idea is that if stake is variable and earnings are linearly proportional to earnings then, mutatis mutandis, it is always more economical for one operator to run just one node with all the stake than several nodes due to the added operational costs. ↩ + +- **extensible and user friendly approach**: the current approach taken by swarm is based on the idea is that, if stake is variable and earnings are linearly proportional to earnings then, mutatis mutandis, it is always more economical for one operator to run just one node with all the stake than several nodes due to the added operational costs. While this has proven to be an effective means to secure the protocol, it has also led to some confusion and it's complexity means it is difficult to develop. The proposed system is intended to be much simpler to reason with and for operators to use and understand. ## Architecture -The balanced neighbourhood assignment (associated with a service) is orchestrated by a smart contract which is deployed together with a [staking contract](https://github.com/ethersphere/storage-incentives/blob/master/src/Staking.sol). The contract API provides 2 transactional endpoints relevant for entry: +The network $S$, comprises the set of all network provider nodes $\{a_0, \ldots, a_n\}$, with cardinality N. + +Each node $a$ has an associated ether address $a_i^\Xi$ and will determine a 32 byte overlay address $a_i^\theta$. + +$ +N = |S|, \quad S = \{a_0, \ldots, a_n\} \quad a_i^{\theta} \in \mathbb{O}_{32}\,,\ a_i^\Xi \in \mathbb{E} +$ + +Then the current depth of the node assignment tree is defined as: + +$ +d_c = \lfloor \log_2(N) \rfloor + 1, \quad d_c \in \mathbb{N} +$ + +The balanced node assignments are orchestrated by a smart contract which will maintain the active node list, their corresponding neighbourhoods, the overlay addresses they have determined, and manage the ingress/egress processes as nodes joini and exit the set of active network service providers. + +This contract is deployed together with a staking contract similar to the [swarm storage incentive staking contract](https://github.com/ethersphere/storage-incentives/blob/master/src/Staking.sol). This contract will retain the total stake treasury, as well as enabling a node operator to deposit, withdrawal and maintain their stake. Concerns should be strictly separated to improve security of locked funds and upgradability of both contracts. + +The node assignment contract is composed of several transactional endpoints: + +__Commit :__ Initially, a node's ether address $a^\Xi$ is registered in the *committers' list* $C$ that records nodes' commitment to participate as a provider in the service network, depositing their application fee ${\$_a}$, which is non-refundable. + +__Get Assigned Overlay :__ -One registers a node by its ether address ($a^\Xi$) in the *committers' list* $C$ that records nodes' commitment to participate as a provider in the associated decentralised service network. + This function includes a read-only call that takes as argument a node's ether address $a^\Xi$ and returns the neighbourhood that the node is currently assigned to. This call is public so that the client can enquire about the neighbourhood they are assigned to -- so that they can mine an overlay address into it, ie., find a nonce that is needed to generate the overlay address. -The other one is called by the staking contract after the service network stake is deposited with a valid nonce, i.e., one that will put the node in the right neighbourhood. This call will place the node among the active node set for the service, and removes the entry from the committers list. This function includes a read-only call that takes as argument a node's ether address and returns a neighbourhood the node is assigned to. This call is public so that the client can enquire about the neighbourhood they are assigned to -- so that they can mine an overlay address into it, ie., find a nonce that is needed to generate the overlay address. +__Assign Overlay:__ One is called by the staking contract, after the service network stake has been deposited with a valid nonce $\vartheta$, i.e., one that, when it is used as a parameter with the Swarm Overlay address calculation $\mathcal{O}$, will produce an overlay address $a^\theta$ from the nodes corresponding ether address $a^\Xi$ which is in the correct neighbourhood. + +$\vartheta \in \mathbb{Z}_{\geq 0} , \quad \mathcal{O}(a,\vartheta) \equiv a^\theta $ + +This call will place the node among the active node set for the service, and removes the entry from the *committers list*. ## Specification @@ -43,24 +70,27 @@ An initially empty list (*committers' list*) of *entry struct* types holds the c + #### Deposit -In order for a node to get its address registered, an amount of $S_0$ (stake zero) must be deposited. +In order for a node to get its address registered, an amount of ${\$_a}$ must be deposited which is non-refundable. #### Uniqueness In order to prevent repeated trials, each node must be registered only once. -After checking the deposit amount and the uniqueness check on the ether address, the current blockheight is recorded with the address by pushing the entry struct ($e_n=$) to the end of committers list. +After checking the deposit amount and the uniqueness check on the ether address, the current blockheight is recorded with the address by pushing the entry struct ($e_i\ = \ $) to the end of committers list. #### Validity -The entry is valid for a period of $B$ blocks after the registration[^55]. +The entry is valid for a period of $B, \quad B < 256 $ blocks after the registration. -[^55]: $B$ is less than $256$, the number of blocks available from within the EVM). +$B$ must be less than $256$, the number of blocks for which the blockhash is available from within the EVM. ((unless the blockhash is recorded)) -Since the blockheight values of the list items are monotonically increasing, entries at the beginning of the list expire first. By iterating upto the first valid entry, expired entries can be iterated on efficiently. +Since the blockheight values of the list items are monotonically increasing, entries at the beginning of the list expire first. By iterating upto the first valid entry, expired entries can be iterated on efficiently. + +>> SIG//NOTE should have time limit on commiting to assigned neighbourhood overlay before providingn nonce. this queing has some unintended consequences in terms of enabling squatting or blocking. the economic disincentives must be calculated and parameters/constants and/or slashing formula created i.e. how much to probably block one neighbourhood then subsequently provoke a split causing data loss. ### Expiry -This function call iterates through all expired entries, burns their deposit, and, by setting the head of the list to the first valid item, removes them from the committer's list. +This function call iterates through all expired entries, burns their deposit, and, by setting the head of the list to the first valid item, removes them from the committer's list. This is called by the assign function (itself called by the staking contract) before the read only call checking if the resulting overlay address falls into the neighbourhood that the registrant was assigned to, i.e., the correctness of the nonce submitted from the perspective of the staking contract. @@ -71,11 +101,14 @@ After calling expiry, the validity of the registration is checked by finding the #### Initialisation -Let $\overline{N}=2^d$ be the lowest power of 2 that is greater than $N$, the number of already assigned registrants, and let $d$ be its exponent ($d=int(log_2(N))+1$). Let $R$ be the array of the remaining unassigned neighbourhoods of this level (i.e., $len(R)=\overline{N}-N$). Whenever $len(R)$ drops to $0$, $d$ is incremented and $\overline{N}=2^d$ adjusts. At any point in time, a $uint256$ array of length $\overline{N}$ is maintained, called the *assignments list* $A$ holding the registered nodes' overlay addresses. -Whenever $d$ changes, new arrays for assigments $A_d$ and remainders $R_d$ are created (both twice the size of the previous one $A_{d-1}$ and $R_{d-1}$). We iterate through the current array and for each overlay address $a^O_i$ at position $i$ copy $a^O_i$ to position $2\cdot i +b_i$ where $b_i=1$ if the $d$-th bit of the address is set: +Let $\overline{N}=2^d$ be the lowest power of 2 that is greater than $N$, the number of already assigned registrants, where $d$ and $N$ are defined as before. Let $R$ be the array of the remaining unassigned neighbourhoods of this level (i.e., $|R|=\overline{N}-N$). + +Whenever the number of slots at this depth, $|R|$ drops to $0$, $d$ is incremented and $\overline{N}=2^d$ adjusts. At any point in time, a $uint256$ array of length $\overline{N}$ is maintained, called the *assignments list* $A$ holding the currently registered nodes' overlay addresses. + +Whenever $d$ changes, new arrays for assigments $A_d$ and remainders $R_d$ are created (both twice the size of the previous one $A_{d-1}$ and $R_{d-1}$). We iterate through the current array and for each overlay address $a_i^\theta$ at position $i$ copy $a_i^\theta$ to position $2i +b_i$ where $b_i=1$ if the $d$-th bit of the address is set: $$ -b_i=a^O_i[d/8] +b_i=a^\theta_i[d/8] $$ $$ @@ -90,37 +123,38 @@ $$ In fact, $A_d$ stands for nodes of a binary tree on level $d$ and the value at each position is the overlay address filling that neighbourhood. When $d$ is incremented, we will need to fill some of the neighbourhoods with the existing nodes. Each node will fill the left child node (when $b_i=0$) or the right one ($b_i=1$) depending on the subsequent bit in their overlay. $$ -\forall 0\leq i < 2^d, A_d[2\cdot i+b_i]=A_{d-1}[i] +\forall\ 0\leq i < 2^d,\quad A_d[2i+b_i]=A_{d-1}[i] $$ As we are filling the assignment list, we know that the whenever a neighbourhood is filled with an already existing node, its sister neighbourhood will be unassigned, therefore we can just record those in the remaining list. $$ -\forall 0\leq i < len(R_d), R_d[i]=2\cdot i+1-b_i +\forall\ 0\leq i < |R_d|,\quad R_d[i]=2i+1-b_i $$ #### Random seed -This internal read-only call takes as its single argument an ether address ($a^\Xi$) and returns a random $uint256$. -After checking if the ether address is a valid registrant by finding the corresponding first (unique) $entry$ struct,[^33] the `difficulty` (randao) is called on the block subsequent to the blockheight registered. Append to it the ether address and hash it to yield what will serve as the random seed for this provider.[^4] +This internal read-only call takes as its single argument an ether address ($a^\Xi$) and returns a (?) random $\rho \in \mathbb{Z}_\text{uint256}$. +After checking if the ether address is a valid registrant by finding the corresponding first (unique) $entry$ struct,[^33] the `difficulty` (randao) is called on the block subsequent to the blockheight registered. Append to it the ether address and hash using $H$ it to yield what will serve as the random seed for this provider.[^4] $$ -\sigma:=\mathit{blockAtHeight}(\mathit{entry}(a^\Xi).\mathit{height}+1).\mathit{difficulty}() +\sigma=\mathit{blockAtHeight}(\mathit{entry}(a^\Xi).\mathit{height}+1).\mathit{difficulty}() $$ $$ -\varrho:=\mathit{uint256}(H(\sigma|a^\Xi)) +\varrho=\mathit{uint256}(H(\sigma|a^\Xi)) $$ -[^33]: Since expiry is not necessarily called when the random seed it called, the blockheight needs to be checked: 1) if block after $h$ (the one in which the registration happened) is available, 2) that the height is not greater than $h+D$ with $D$ being the validity period in blocks. +[^33]: Since expiry is not necessarily called when the random seed it called, the blockheight needs to be checked: 1) if block after $h$ (the one in which the registration happened) is available, 2) that the height is not greater than $h+B$ with $B$ being the validity period in blocks. [^4]: Even if on a POA chain, and no randao, this seed cannot be known to the registering provider and their colluding associates, but nonetheless should be deterministic once its set. #### Neighbourhood -This component must be available as a public readonly endpoint taking a node's ($n$) ether address ($a^\Xi_n$) as a single argument. -The random nonce $\varrho_n$ is used to select a neighbourhood $nh$ for a provider from the remaining unassigned neighbourhood list of level $d$: +This component must be available as a public readonly endpoint taking a node's ($j\text{-th}$) ether address ($a^\Xi_j$) as a single argument. + +A random nonce $\varrho_n$ is used to select a neighbourhood $nh$ for a provider from the remaining unassigned neighbourhood list of level $d$: $$ i:=\varrho\mod len(R_d) @@ -154,6 +188,9 @@ $$ - if $R_d$ is now of zero length, the $d$ is incremented, and new assignment and remaining lists are initialised as per section 'initialisation' above. - provider's entry is removed from the committers' list. +### Economic Analysis + + ### Further endpoints From 54c1fd2096437011dceeffe92d53b78a883c8649 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Sat, 21 Mar 2026 08:32:46 +0100 Subject: [PATCH 13/23] Revise SWIP-39 for clarity and detail enhancement (#87) Refine the SWIP-39 document to clarify the protocol mechanism for balanced neighbourhood registry and node assignment. Enhance sections on architecture, model, and data structure to improve understanding of the system's operational principles. --- SWIPs/swip-39.md | 479 +++++++++++++---------------------------------- 1 file changed, 135 insertions(+), 344 deletions(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index d48749f1..930ff0cd 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -11,9 +11,15 @@ created: 2025-07-21 # Balanced Neighbourhood Registry aka Smart Neighbourhood Management + ## Abstract -This SWIP introduces a systematic way for operators to enter the network in such a way that their overlay addresses are balanced within the address space. Importantly, operators must register their commitment to take part and on doing so will be assigned a random neighbourhood from the queue of those next to be occupied. Since a joining node operator is unable to position themselves at a specific point in the network without a significant time or financial penalty, attack vectors that rely on this are rendered infeasible. +This proposal defines a protocol mechanism that enforces uniform distribution of nodes across the Swarm address space while preventing adversarial positioning. Nodes commit to participation and are assigned neighbourhoods through delayed entropy derived from on-chain randomness. The assignment is therefore unpredictable at commit time and reproducible at validation time. Since a joining node operator is unable to position themselves at a specific point in the network without a significant time or financial penalty, attack vectors that rely on this are rendered infeasible. + + +The system maintains a structural invariant over the address space ensuring that no prefix of length $d-1$ is empty. This invariant is preserved through strictly local operations, namely assignment, deregistration, and rebalance, without requiring global restructuring. The design achieves fairness, bounded operational cost, and resistance to manipulation. + + ## Motivation @@ -29,424 +35,209 @@ There are multiple considerations that motivate such a scheme: - **extensible and user friendly approach**: the current approach taken by swarm is based on the idea is that, if stake is variable and earnings are linearly proportional to earnings then, mutatis mutandis, it is always more economical for one operator to run just one node with all the stake than several nodes due to the added operational costs. While this has proven to be an effective means to secure the protocol, it has also led to some confusion and it's complexity means it is difficult to develop. The proposed system is intended to be much simpler to reason with and for operators to use and understand. -## Architecture +A decentralised service network assumes that workload is distributed according to a uniform random process over the address space. This assumption only holds if the distribution of nodes itself is approximately uniform. In practice, allowing operators to influence their placement leads to clustering and adversarial positioning, while global rebalancing mechanisms are impractical in a smart contract setting due to their cost and coordination requirements. -The network $S$, comprises the set of all network provider nodes $\{a_0, \ldots, a_n\}$, with cardinality N. +The present design replaces behavioural assumptions with structural guarantees. Node placement is determined externally by the protocol, and the system continuously enforces coverage through a local invariant that is preserved under all admissible transitions. -Each node $a$ has an associated ether address $a_i^\Xi$ and will determine a 32 byte overlay address $a_i^\theta$. +--- -$ -N = |S|, \quad S = \{a_0, \ldots, a_n\} \quad a_i^{\theta} \in \mathbb{O}_{32}\,,\ a_i^\Xi \in \mathbb{E} -$ +## Model and Notation -Then the current depth of the node assignment tree is defined as: +Let the set of active nodes be denoted by -$ -d_c = \lfloor \log_2(N) \rfloor + 1, \quad d_c \in \mathbb{N} -$ +$$ +S = \{a_0, \ldots, a_n\}, \quad N = |S|. +$$ -The balanced node assignments are orchestrated by a smart contract which will maintain the active node list, their corresponding neighbourhoods, the overlay addresses they have determined, and manage the ingress/egress processes as nodes joini and exit the set of active network service providers. +Each node is identified by an Ethereum address $a^\Xi$ and an overlay address $a^\theta \in \mathbb{O}_{32}$. -This contract is deployed together with a staking contract similar to the [swarm storage incentive staking contract](https://github.com/ethersphere/storage-incentives/blob/master/src/Staking.sol). This contract will retain the total stake treasury, as well as enabling a node operator to deposit, withdrawal and maintain their stake. Concerns should be strictly separated to improve security of locked funds and upgradability of both contracts. +The system maintains a depth parameter $d \in \mathbb{N}$ such that -The node assignment contract is composed of several transactional endpoints: +$$ +2^{d-1} < N \le 2^d. +$$ -__Commit :__ Initially, a node's ether address $a^\Xi$ is registered in the *committers' list* $C$ that records nodes' commitment to participate as a provider in the service network, depositing their application fee ${\$_a}$, which is non-refundable. +Neighbourhoods are represented as indices in a complete binary space: -__Get Assigned Overlay :__ +$$ +I : \{0, \ldots, 2^d - 1\} \to \mathbb{O}_{32} \cup \{0\}, +$$ - This function includes a read-only call that takes as argument a node's ether address $a^\Xi$ and returns the neighbourhood that the node is currently assigned to. This call is public so that the client can enquire about the neighbourhood they are assigned to -- so that they can mine an overlay address into it, ie., find a nonce that is needed to generate the overlay address. +where $I[i] = 0$ denotes an empty slot. A reverse mapping -__Assign Overlay:__ One is called by the staking contract, after the service network stake has been deposited with a valid nonce $\vartheta$, i.e., one that, when it is used as a parameter with the Swarm Overlay address calculation $\mathcal{O}$, will produce an overlay address $a^\theta$ from the nodes corresponding ether address $a^\Xi$ which is in the correct neighbourhood. +$$ +J : a^\Xi \mapsto i +$$ -$\vartheta \in \mathbb{Z}_{\geq 0} , \quad \mathcal{O}(a,\vartheta) \equiv a^\theta $ +associates each node with its assigned index. -This call will place the node among the active node set for the service, and removes the entry from the *committers list*. +For each index $i$, define the pair -## Specification +$$ +\mathcal{P}_i = \{2i, 2i+1\}. +$$ -### Registration -An initially empty list (*committers' list*) of *entry struct* types holds the current committers. The struct holds information about the ether address of the node, the blockheight the address registered at. - +--- +## Structural Invariant +The system enforces the condition -#### Deposit -In order for a node to get its address registered, an amount of ${\$_a}$ must be deposited which is non-refundable. +$$ +\forall i, \quad I[2i] \neq 0 \;\lor\; I[2i+1] \neq 0, +$$ -#### Uniqueness -In order to prevent repeated trials, each node must be registered only once. +which ensures that every prefix of length $d-1$ contains at least one node. This invariant defines the admissible states of the system. -After checking the deposit amount and the uniqueness check on the ether address, the current blockheight is recorded with the address by pushing the entry struct ($e_i\ = \ $) to the end of committers list. +--- -#### Validity -The entry is valid for a period of $B, \quad B < 256 $ blocks after the registration. +## Data Structure -$B$ must be less than $256$, the number of blocks for which the blockhash is available from within the EVM. ((unless the blockhash is recorded)) +The assignment structure is implemented as an implicit complete binary trie over the index space. Each node $v$ of the trie corresponds to a contiguous interval of indices and maintains two quantities. -Since the blockheight values of the list items are monotonically increasing, entries at the beginning of the list expire first. By iterating upto the first valid entry, expired entries can be iterated on efficiently. +The first quantity $F(v)$ denotes the number of free slots in the subtree rooted at $v$. Formally, if $L(v)$ denotes the set of leaf indices under $v$, then ->> SIG//NOTE should have time limit on commiting to assigned neighbourhood overlay before providingn nonce. this queing has some unintended consequences in terms of enabling squatting or blocking. the economic disincentives must be calculated and parameters/constants and/or slashing formula created i.e. how much to probably block one neighbourhood then subsequently provoke a split causing data loss. +$$ +F(v) = |\{ i \in L(v) \mid I[i] = 0 \}|. +$$ -### Expiry +The second quantity $E(v)$ denotes the number of indices $i$ in the subtree such that both elements of $\mathcal{P}_i$ are occupied. These correspond to candidate donor pairs. -This function call iterates through all expired entries, burns their deposit, and, by setting the head of the list to the first valid item, removes them from the committer's list. +Both quantities satisfy recursive relations -This is called by the assign function (itself called by the staking contract) before the read only call checking if the resulting overlay address falls into the neighbourhood that the registrant was assigned to, i.e., the correctness of the nonce submitted from the perspective of the staking contract. +$$ +F(v) = F(v_L) + F(v_R), \quad E(v) = E(v_L) + E(v_R), +$$ -### Assign +where $v_L$ and $v_R$ denote the left and right children of $v$. Updates propagate along the path from a leaf to the root, resulting in logarithmic complexity. -The assign call is the second transactional endpoint called by the staking contract. It takes the provider's ether address and as well as the mined overlay as arguments. -After calling expiry, the validity of the registration is checked by finding the entry for the ether address in the committers' list. +--- -#### Initialisation +## Data Structure Illustration + +```mermaid +graph TD + R((root)) + R --> A + R --> B + A --> A1 + A --> A2 + B --> B1 + B --> B2 +``` -Let $\overline{N}=2^d$ be the lowest power of 2 that is greater than $N$, the number of already assigned registrants, where $d$ and $N$ are defined as before. Let $R$ be the array of the remaining unassigned neighbourhoods of this level (i.e., $|R|=\overline{N}-N$). +Each leaf corresponds to an index. Internal nodes aggregate subtree quantities $F$ and $E$. -Whenever the number of slots at this depth, $|R|$ drops to $0$, $d$ is incremented and $\overline{N}=2^d$ adjusts. At any point in time, a $uint256$ array of length $\overline{N}$ is maintained, called the *assignments list* $A$ holding the currently registered nodes' overlay addresses. +--- -Whenever $d$ changes, new arrays for assigments $A_d$ and remainders $R_d$ are created (both twice the size of the previous one $A_{d-1}$ and $R_{d-1}$). We iterate through the current array and for each overlay address $a_i^\theta$ at position $i$ copy $a_i^\theta$ to position $2i +b_i$ where $b_i=1$ if the $d$-th bit of the address is set: +## Entropy -$$ -b_i=a^\theta_i[d/8] -$$ +A node that registers at block height $h$ derives its randomness from $$ -b_i\gg=7-(d\mod 8) +\rho = H(\text{blockhash}(h+1) \parallel a^\Xi \parallel h), $$ +which is not known at the time of registration. The validity window $B < 256$ ensures that the referenced blockhash remains accessible. -$$ -b_i{\land\hspace{-3pt}=}1 -$$ - -In fact, $A_d$ stands for nodes of a binary tree on level $d$ and the value at each position is the overlay address filling that neighbourhood. When $d$ is incremented, we will need to fill some of the neighbourhoods with the existing nodes. Each node will fill the left child node (when $b_i=0$) or the right one ($b_i=1$) depending on the subsequent bit in their overlay. +--- -$$ -\forall\ 0\leq i < 2^d,\quad A_d[2i+b_i]=A_{d-1}[i] -$$ +## Assignment -As we are filling the assignment list, we know that the whenever a neighbourhood is filled with an already existing node, its sister neighbourhood will be unassigned, therefore we can just record those in the remaining list. +Let $M = 2^d - N$ denote the number of free slots. A node computes $$ -\forall\ 0\leq i < |R_d|,\quad R_d[i]=2i+1-b_i +k = \rho \bmod M. $$ +The assigned index is determined by descending the trie. At a node $v$, let $F(v_L)$ denote the number of free slots in the left subtree. If $k < F(v_L)$, the traversal continues to the left child. Otherwise, the traversal continues to the right child with updated rank $k \leftarrow k - F(v_L)$. This procedure terminates at a leaf index $i$ such that $I[i] = 0$. -#### Random seed +--- -This internal read-only call takes as its single argument an ether address ($a^\Xi$) and returns a (?) random $\rho \in \mathbb{Z}_\text{uint256}$. -After checking if the ether address is a valid registrant by finding the corresponding first (unique) $entry$ struct,[^33] the `difficulty` (randao) is called on the block subsequent to the blockheight registered. Append to it the ether address and hash using $H$ it to yield what will serve as the random seed for this provider.[^4] +## Deregistration and Rebalancing -$$ -\sigma=\mathit{blockAtHeight}(\mathit{entry}(a^\Xi).\mathit{height}+1).\mathit{difficulty}() -$$ +Consider a node occupying index $r$. Let $i = \lfloor r/2 \rfloor$ be the corresponding pair index. -$$ -\varrho=\mathit{uint256}(H(\sigma|a^\Xi)) -$$ +If the sibling index in $\mathcal{P}_i$ is occupied, removal proceeds directly and the invariant remains satisfied. -[^33]: Since expiry is not necessarily called when the random seed it called, the blockheight needs to be checked: 1) if block after $h$ (the one in which the registration happened) is available, 2) that the height is not greater than $h+B$ with $B$ being the validity period in blocks. +If removal would leave $\mathcal{P}_i$ empty, a rebalance is required. A donor pair is selected using the same rank-based traversal over $E(v)$. From the selected pair, one of the two nodes is chosen and removed. The donor node is reinserted into the commit queue and assigned to the empty pair. -[^4]: Even if on a POA chain, and no randao, this seed cannot be known to the registering provider and their colluding associates, but nonetheless should be deterministic once its set. +The original node is removed only after the donor successfully completes reassignment, ensuring that the invariant is never violated. -#### Neighbourhood +--- -This component must be available as a public readonly endpoint taking a node's ($j\text{-th}$) ether address ($a^\Xi_j$) as a single argument. +## Depth Reduction -A random nonce $\varrho_n$ is used to select a neighbourhood $nh$ for a provider from the remaining unassigned neighbourhood list of level $d$: +If $N = 2^{d-1}$, every pair contains exactly one node and no donor exists. In this case, the representation is reduced by setting $d \leftarrow d-1$ and mapping each pair to a single index. This transformation is deterministic because no pair contains more than one node. -$$ -i:=\varrho\mod len(R_d) -$$ +--- -$$ -nh=R[i] -$$ +## Interface -#### Checking the overlay +| Function | Description | +|----------|------------| +| Register(a) | Records commitment and blockheight | +| GetPrefix(a) | Returns assigned index | +| Validate(a,o) | Verifies overlay matches assigned index | +| Insert(a,o) | Inserts node and updates structure | +| Deregister(a) | Initiates removal | +| Remove(i) | Removes node from index | +| Rebalance(i) | Repairs invariant | -The overlay (obtained by mining the nonce) is checked to fall in the correct neighbourhood r: -The check validates the address $a^O_n$ if and only if: +--- -$$ -r=a^O_n\gg(255-d) -$$ +## Sequence Diagram + +```mermaid +sequenceDiagram + participant Client + participant Registry + participant Trie + + Client->>Registry: Register(a) + Registry-->>Client: prefix + Client->>Registry: Validate(a,o) + Registry->>Trie: update(F,E) + Client->>Registry: Deregister(a) + Registry->>Trie: check invariant + Registry->>Trie: select donor + Registry->>Trie: update +``` -#### Assignment +--- -If the overlay check passes, +## Economic Considerations -- the nodes' overlay address is assigned to a neighbourhood of depth $d$. +Fairness of the system requires that each node is associated with a fixed stake. Under this assumption, total stake is proportional to the number of nodes, $$ -A_d[r]=a^O_n +\text{Total stake} \propto N, $$ -- $N$ is incremented -- the $i$-th item is removed from remaining open neighbourhood list $R_d$. -- if $R_d$ is now of zero length, the $d$ is incremented, and new assignment and remaining lists are initialised as per section 'initialisation' above. -- provider's entry is removed from the committers' list. +and no operator gains advantage by concentrating stake into fewer nodes. This aligns economic incentives with the structural invariant. -### Economic Analysis +--- +## Impact +The system achieves uniform distribution of nodes, resistance to adversarial placement, and predictable operational costs. Rebalancing introduces a dependency on donor behaviour; however, the resulting delay is bounded and practically absorbed by the off-boarding process, which already operates as a queue. Consequently, rebalance latency does not introduce observable instability. -### Further endpoints +--- -A public read-only endpoint exists for querying neighbourhoods as well as nodes. Accessor for $d$ and $N$ will return the current neighbourhood depth and the current number of assigned neighbourhoods. A public accessor for $A_d$ will return for a neighbourhood (between $0$ and $2^d-1 inclusive) the overlay of the node assigned to that neighbourhood. Another endpoint will return for any overlay $o$ the closest node, so that the network service can find responsible nodes for any task with address in the space shared by overlays: +## Security Analysis -$$ -g(a)=A_d[a\gg(255-d)] -$$ +The commit-and-delay entropy mechanism prevents nodes from predicting or influencing their placement. Structural constraints prevent clustering, making Sybil attacks costly and ineffective. Validator influence is limited to a single block and does not provide sufficient control to bias assignment. -### Deregistration +Rebalancing cannot be blocked as long as $N > 2^{d-1}$, since a donor must exist. If a donor fails to complete reassignment, expiry ensures that another donor is selected, guaranteeing progress. -Only called from the Staking contract, deregister deletes the entry for the neighbourhood belonging to the given address, makes the neighbourhood available in $R$ +--- +## Testing -## Implementation notes - -### Changes to the staking contract - -### Changes to the bee client - -A new endpoint to bee client must be added to register a node that is not yet registered to be assigned a neighbourhood. Once the neighbourhood is known, the client can mine the nonce needed to place the overlay in the required neighbourhood. - -### Migration - -Since a new updated staking contract, a stake migration will be needed for the upgrade. Before the change, all the simplification of the staking contract is recommended, especially to allow fixed stake in order to realign redundancy -of storage and monetary incentive: with a fixed amount staked, total stake is linearly proportional to the number of nodes, and therefore comparisons across neighbourhoods can be made based on the number of nodes. In particular, the arbitrary balanced assignment makes sense in terms of incentives (expected revenue). - -### Putting a node in each neighbourhood. - -## Contract - -```sol -// SPDX-License-Identifier: MIT -pragma solidity ^0.8.19; - -contract BalancedNeighbourhoodRegistry { - // -------------------- - // Configurable values - // -------------------- - uint256 public constant STAKE = 10,000,000,000,000,000 wei; // Stake required - uint256 public constant VALID_FOR = 128; // Validity window in blocks - - // -------------------- - // Structs and Storage - // -------------------- - struct Entry { - address committer; - uint256 height; - } - - Entry[] public committers; // List of committers (registered but as yet unassigned nodes) - mapping(address => bool) public hasCommitted; // Track if an address has committed - - uint256 public N; // Number of assigned nodes - uint256 public d = 1; // Neighbourhood depth - uint256 public currentPower = 2; // 2^d - - // Assignments: overlay address for neighbourhoods of depth d - bytes32[] public A; - - // Remaining unassigned neighbourhoods of depth d - uint256[] public R; - - // -------------------- - // Events - // -------------------- - event Registered(address indexed node, uint256 blockHeight); - event Assigned(address indexed node, uint256 neighbourhood, bytes32 overlay); - event DepthUpgraded(uint256 newDepth); - event Unassigned(address indexed node, uint256 neighbourhood); - - // -------------------- - // Registration endpoint - // -------------------- - function register() external payable { - require(msg.value == STAKE, "Invalid stake"); - require(!hasCommitted[msg.sender], "Already registered"); - - committers.push(Entry(msg.sender, block.number)); - hasCommitted[msg.sender] = true; - - emit Registered(msg.sender, block.number); - } - - // -------------------- - // Expire old entries - // -------------------- - function _expire() internal { - while (committers.length > 0) { - Entry storage e = committers[0]; - if (block.number <= e.height + VALID_FOR) { - break; - } - hasCommitted[e.committer] = false; - _removeCommitter(0); - // Burn logic: funds stay locked. - } - } - - // -------------------- - // Remove committer from the list - // -------------------- - // This function is used internally to remove a committer from the list. - // It shifts the elements to the left and pops the last element. - function _removeCommitter(uint index) internal { - if (index >= committers.length) return; - - for (uint i = index; i < committers.length - 1; i++) { - committers[i] = committers[i + 1]; - } - committers.pop(); - } - - // -------------------- - // Find entry for a committer in the committer list - // Returns the index of the entry if found, otherwise reverts. - // -------------------- - function _findEntryFor(address _a) internal view returns (uint) { - for (uint i = 0; i < committers.length; i++) { - if (committers[i].committer == _a) { - return i; - } - } - revert("Not registered"); - } - - // -------------------- - // Randomness - // -------------------- - function _randomSeed(address _a) internal view returns (uint256) { - uint i = _findEntryFor(_a); - uint256 h = committers[i].height; - // Ensure the block number is valid as expire may not have been called - require(block.number > h + 1, "Too early"); - require(block.number <= h + VALID_FOR, "Registration expired"); - // Use blockhash to generate a random seed - bytes32 bh = blockhash(h + 1); - return uint256(keccak256(abi.encodePacked(bh, _a))); - } - - // -------------------- - // Public View: Neighbourhood - // -------------------- - function getNeighbourhood(address _a) public view returns (uint256) { - uint256 r = _randomSeed(_a); - require(R.length > 0, "No available neighbourhoods"); - return R[r % R.length]; - } - - // -------------------- - // Assign node overlay to neighbourhood - // -------------------- - function assign(address _a, bytes32 _overlay) external { - _expire(); - - // Check registration - uint256 nh = getNeighbourhood(_a); - uint256 overlayNh = uint256(_overlay) >> (256 - d); - - require(overlayNh == nh, "Overlay doesn't match neighbourhood"); - - A[nh] = _overlay; - N++; - - // Remove neighbourhood from R - _removeFromR(nh); - - // Check if R is empty - if (R.length == 0) { - _upgradeDepth(); - } - - hasCommitted[_a] = false; - _removeEntry(_a); - - emit Assigned(_a, nh, _overlay); - } - - //---------------------- - // unregister - //---------------------- - function unregister(address _a, bytes32 _overlay) external { - uint nh = uint256(_overlay) >> (256 - d); - A[nh] = bytes32(0); - R.push(nh); - N--; - // return funds to the committer - payable(_a).transfer(STAKE); - emit Unassigned(_a, nh); - } - - // -------------------- - // Internal functions to manage R and committers - // -------------------- - function _removeFromR(uint256 nh) internal { - for (uint i = 0; i < R.length; i++) { - if (R[i] == nh) { - // replace the removed element with the last element and pop - R[i] = R[R.length - 1]; - R.pop(); - return; - } - } - } - - function _removeEntry(address _a) internal { - uint i = _findEntryFor(_a); - require(i < committers.length, "Entry not found"); - _removeCommitter(i); - } - - // -------------------- - // Expand A and R when needed - // -------------------- - function _upgradeDepth() internal { - currentPower = 2 ** d; - - delete R; - for (uint i = 0; i < currentPower; i++) { - R.push(0); - // Ensure A has enough space for the new neighbourhoods); - A.push(bytes32(0)); - } - for (uint i = currentPower - 1; i < currentPower; i--) { - uint b = uint256(A[i]) >> (256 - d) % 2; - A[2*i+b] = A[i]; - uint256 j = 2*i+1-b; - if (2 * j < currentPower) { - A[j] = bytes32(0); // Clear the old address - } - R[i] = j; - } - d++; - require(d <= 32, "Maximum depth exceeded"); - emit DepthUpgraded(d); - } - - // -------------------- - // Accessors - // -------------------- - function getDepth() external view returns (uint) { - return d; - } - - function getN() external view returns (uint256) { - return N; - } - - function getOverlayForNeighbourhood(uint256 nh) external view returns (bytes32) { - return A[nh]; - } - - function getClosestNode(bytes32 key) external view returns (bytes32) { - uint256 prefix = uint256(key) >> (256 - d); - return A[prefix]; - } -} -``` +Correctness requires verifying that the invariant is preserved under all state transitions. Simulation should confirm that assignment remains uniform, that rebalancing converges, and that depth transitions occur at the correct thresholds. + +--- + +## Conclusion +This proposal replaces global restructuring with local invariant enforcement over a compact data structure. It achieves fairness, efficiency, and robustness through deterministic rules and probabilistic assignment, making it suitable for large-scale decentralised operation. From ce60560c6f967a52ac84b2f13127a4a2206cfd13 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Sat, 21 Mar 2026 08:51:04 +0100 Subject: [PATCH 14/23] Revert "Revise SWIP-39 for clarity and detail enhancement (#87)" (#89) This reverts commit 54c1fd2096437011dceeffe92d53b78a883c8649. --- SWIPs/swip-39.md | 479 ++++++++++++++++++++++++++++++++++------------- 1 file changed, 344 insertions(+), 135 deletions(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index 930ff0cd..d48749f1 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -11,15 +11,9 @@ created: 2025-07-21 # Balanced Neighbourhood Registry aka Smart Neighbourhood Management - ## Abstract -This proposal defines a protocol mechanism that enforces uniform distribution of nodes across the Swarm address space while preventing adversarial positioning. Nodes commit to participation and are assigned neighbourhoods through delayed entropy derived from on-chain randomness. The assignment is therefore unpredictable at commit time and reproducible at validation time. Since a joining node operator is unable to position themselves at a specific point in the network without a significant time or financial penalty, attack vectors that rely on this are rendered infeasible. - - -The system maintains a structural invariant over the address space ensuring that no prefix of length $d-1$ is empty. This invariant is preserved through strictly local operations, namely assignment, deregistration, and rebalance, without requiring global restructuring. The design achieves fairness, bounded operational cost, and resistance to manipulation. - - +This SWIP introduces a systematic way for operators to enter the network in such a way that their overlay addresses are balanced within the address space. Importantly, operators must register their commitment to take part and on doing so will be assigned a random neighbourhood from the queue of those next to be occupied. Since a joining node operator is unable to position themselves at a specific point in the network without a significant time or financial penalty, attack vectors that rely on this are rendered infeasible. ## Motivation @@ -35,209 +29,424 @@ There are multiple considerations that motivate such a scheme: - **extensible and user friendly approach**: the current approach taken by swarm is based on the idea is that, if stake is variable and earnings are linearly proportional to earnings then, mutatis mutandis, it is always more economical for one operator to run just one node with all the stake than several nodes due to the added operational costs. While this has proven to be an effective means to secure the protocol, it has also led to some confusion and it's complexity means it is difficult to develop. The proposed system is intended to be much simpler to reason with and for operators to use and understand. -A decentralised service network assumes that workload is distributed according to a uniform random process over the address space. This assumption only holds if the distribution of nodes itself is approximately uniform. In practice, allowing operators to influence their placement leads to clustering and adversarial positioning, while global rebalancing mechanisms are impractical in a smart contract setting due to their cost and coordination requirements. +## Architecture -The present design replaces behavioural assumptions with structural guarantees. Node placement is determined externally by the protocol, and the system continuously enforces coverage through a local invariant that is preserved under all admissible transitions. +The network $S$, comprises the set of all network provider nodes $\{a_0, \ldots, a_n\}$, with cardinality N. ---- +Each node $a$ has an associated ether address $a_i^\Xi$ and will determine a 32 byte overlay address $a_i^\theta$. -## Model and Notation +$ +N = |S|, \quad S = \{a_0, \ldots, a_n\} \quad a_i^{\theta} \in \mathbb{O}_{32}\,,\ a_i^\Xi \in \mathbb{E} +$ -Let the set of active nodes be denoted by +Then the current depth of the node assignment tree is defined as: -$$ -S = \{a_0, \ldots, a_n\}, \quad N = |S|. -$$ +$ +d_c = \lfloor \log_2(N) \rfloor + 1, \quad d_c \in \mathbb{N} +$ -Each node is identified by an Ethereum address $a^\Xi$ and an overlay address $a^\theta \in \mathbb{O}_{32}$. +The balanced node assignments are orchestrated by a smart contract which will maintain the active node list, their corresponding neighbourhoods, the overlay addresses they have determined, and manage the ingress/egress processes as nodes joini and exit the set of active network service providers. -The system maintains a depth parameter $d \in \mathbb{N}$ such that +This contract is deployed together with a staking contract similar to the [swarm storage incentive staking contract](https://github.com/ethersphere/storage-incentives/blob/master/src/Staking.sol). This contract will retain the total stake treasury, as well as enabling a node operator to deposit, withdrawal and maintain their stake. Concerns should be strictly separated to improve security of locked funds and upgradability of both contracts. -$$ -2^{d-1} < N \le 2^d. -$$ +The node assignment contract is composed of several transactional endpoints: -Neighbourhoods are represented as indices in a complete binary space: +__Commit :__ Initially, a node's ether address $a^\Xi$ is registered in the *committers' list* $C$ that records nodes' commitment to participate as a provider in the service network, depositing their application fee ${\$_a}$, which is non-refundable. -$$ -I : \{0, \ldots, 2^d - 1\} \to \mathbb{O}_{32} \cup \{0\}, -$$ +__Get Assigned Overlay :__ -where $I[i] = 0$ denotes an empty slot. A reverse mapping + This function includes a read-only call that takes as argument a node's ether address $a^\Xi$ and returns the neighbourhood that the node is currently assigned to. This call is public so that the client can enquire about the neighbourhood they are assigned to -- so that they can mine an overlay address into it, ie., find a nonce that is needed to generate the overlay address. -$$ -J : a^\Xi \mapsto i -$$ +__Assign Overlay:__ One is called by the staking contract, after the service network stake has been deposited with a valid nonce $\vartheta$, i.e., one that, when it is used as a parameter with the Swarm Overlay address calculation $\mathcal{O}$, will produce an overlay address $a^\theta$ from the nodes corresponding ether address $a^\Xi$ which is in the correct neighbourhood. -associates each node with its assigned index. +$\vartheta \in \mathbb{Z}_{\geq 0} , \quad \mathcal{O}(a,\vartheta) \equiv a^\theta $ -For each index $i$, define the pair +This call will place the node among the active node set for the service, and removes the entry from the *committers list*. -$$ -\mathcal{P}_i = \{2i, 2i+1\}. -$$ +## Specification ---- +### Registration +An initially empty list (*committers' list*) of *entry struct* types holds the current committers. The struct holds information about the ether address of the node, the blockheight the address registered at. + -## Structural Invariant -The system enforces the condition -$$ -\forall i, \quad I[2i] \neq 0 \;\lor\; I[2i+1] \neq 0, -$$ +#### Deposit +In order for a node to get its address registered, an amount of ${\$_a}$ must be deposited which is non-refundable. -which ensures that every prefix of length $d-1$ contains at least one node. This invariant defines the admissible states of the system. +#### Uniqueness +In order to prevent repeated trials, each node must be registered only once. ---- +After checking the deposit amount and the uniqueness check on the ether address, the current blockheight is recorded with the address by pushing the entry struct ($e_i\ = \ $) to the end of committers list. -## Data Structure +#### Validity +The entry is valid for a period of $B, \quad B < 256 $ blocks after the registration. -The assignment structure is implemented as an implicit complete binary trie over the index space. Each node $v$ of the trie corresponds to a contiguous interval of indices and maintains two quantities. +$B$ must be less than $256$, the number of blocks for which the blockhash is available from within the EVM. ((unless the blockhash is recorded)) -The first quantity $F(v)$ denotes the number of free slots in the subtree rooted at $v$. Formally, if $L(v)$ denotes the set of leaf indices under $v$, then +Since the blockheight values of the list items are monotonically increasing, entries at the beginning of the list expire first. By iterating upto the first valid entry, expired entries can be iterated on efficiently. -$$ -F(v) = |\{ i \in L(v) \mid I[i] = 0 \}|. -$$ +>> SIG//NOTE should have time limit on commiting to assigned neighbourhood overlay before providingn nonce. this queing has some unintended consequences in terms of enabling squatting or blocking. the economic disincentives must be calculated and parameters/constants and/or slashing formula created i.e. how much to probably block one neighbourhood then subsequently provoke a split causing data loss. -The second quantity $E(v)$ denotes the number of indices $i$ in the subtree such that both elements of $\mathcal{P}_i$ are occupied. These correspond to candidate donor pairs. +### Expiry -Both quantities satisfy recursive relations +This function call iterates through all expired entries, burns their deposit, and, by setting the head of the list to the first valid item, removes them from the committer's list. -$$ -F(v) = F(v_L) + F(v_R), \quad E(v) = E(v_L) + E(v_R), -$$ +This is called by the assign function (itself called by the staking contract) before the read only call checking if the resulting overlay address falls into the neighbourhood that the registrant was assigned to, i.e., the correctness of the nonce submitted from the perspective of the staking contract. -where $v_L$ and $v_R$ denote the left and right children of $v$. Updates propagate along the path from a leaf to the root, resulting in logarithmic complexity. +### Assign ---- +The assign call is the second transactional endpoint called by the staking contract. It takes the provider's ether address and as well as the mined overlay as arguments. +After calling expiry, the validity of the registration is checked by finding the entry for the ether address in the committers' list. -## Data Structure Illustration - -```mermaid -graph TD - R((root)) - R --> A - R --> B - A --> A1 - A --> A2 - B --> B1 - B --> B2 -``` +#### Initialisation -Each leaf corresponds to an index. Internal nodes aggregate subtree quantities $F$ and $E$. +Let $\overline{N}=2^d$ be the lowest power of 2 that is greater than $N$, the number of already assigned registrants, where $d$ and $N$ are defined as before. Let $R$ be the array of the remaining unassigned neighbourhoods of this level (i.e., $|R|=\overline{N}-N$). ---- +Whenever the number of slots at this depth, $|R|$ drops to $0$, $d$ is incremented and $\overline{N}=2^d$ adjusts. At any point in time, a $uint256$ array of length $\overline{N}$ is maintained, called the *assignments list* $A$ holding the currently registered nodes' overlay addresses. -## Entropy +Whenever $d$ changes, new arrays for assigments $A_d$ and remainders $R_d$ are created (both twice the size of the previous one $A_{d-1}$ and $R_{d-1}$). We iterate through the current array and for each overlay address $a_i^\theta$ at position $i$ copy $a_i^\theta$ to position $2i +b_i$ where $b_i=1$ if the $d$-th bit of the address is set: -A node that registers at block height $h$ derives its randomness from +$$ +b_i=a^\theta_i[d/8] +$$ $$ -\rho = H(\text{blockhash}(h+1) \parallel a^\Xi \parallel h), +b_i\gg=7-(d\mod 8) $$ -which is not known at the time of registration. The validity window $B < 256$ ensures that the referenced blockhash remains accessible. ---- +$$ +b_i{\land\hspace{-3pt}=}1 +$$ -## Assignment +In fact, $A_d$ stands for nodes of a binary tree on level $d$ and the value at each position is the overlay address filling that neighbourhood. When $d$ is incremented, we will need to fill some of the neighbourhoods with the existing nodes. Each node will fill the left child node (when $b_i=0$) or the right one ($b_i=1$) depending on the subsequent bit in their overlay. + +$$ +\forall\ 0\leq i < 2^d,\quad A_d[2i+b_i]=A_{d-1}[i] +$$ -Let $M = 2^d - N$ denote the number of free slots. A node computes +As we are filling the assignment list, we know that the whenever a neighbourhood is filled with an already existing node, its sister neighbourhood will be unassigned, therefore we can just record those in the remaining list. $$ -k = \rho \bmod M. +\forall\ 0\leq i < |R_d|,\quad R_d[i]=2i+1-b_i $$ -The assigned index is determined by descending the trie. At a node $v$, let $F(v_L)$ denote the number of free slots in the left subtree. If $k < F(v_L)$, the traversal continues to the left child. Otherwise, the traversal continues to the right child with updated rank $k \leftarrow k - F(v_L)$. This procedure terminates at a leaf index $i$ such that $I[i] = 0$. ---- +#### Random seed -## Deregistration and Rebalancing +This internal read-only call takes as its single argument an ether address ($a^\Xi$) and returns a (?) random $\rho \in \mathbb{Z}_\text{uint256}$. +After checking if the ether address is a valid registrant by finding the corresponding first (unique) $entry$ struct,[^33] the `difficulty` (randao) is called on the block subsequent to the blockheight registered. Append to it the ether address and hash using $H$ it to yield what will serve as the random seed for this provider.[^4] -Consider a node occupying index $r$. Let $i = \lfloor r/2 \rfloor$ be the corresponding pair index. +$$ +\sigma=\mathit{blockAtHeight}(\mathit{entry}(a^\Xi).\mathit{height}+1).\mathit{difficulty}() +$$ -If the sibling index in $\mathcal{P}_i$ is occupied, removal proceeds directly and the invariant remains satisfied. +$$ +\varrho=\mathit{uint256}(H(\sigma|a^\Xi)) +$$ -If removal would leave $\mathcal{P}_i$ empty, a rebalance is required. A donor pair is selected using the same rank-based traversal over $E(v)$. From the selected pair, one of the two nodes is chosen and removed. The donor node is reinserted into the commit queue and assigned to the empty pair. +[^33]: Since expiry is not necessarily called when the random seed it called, the blockheight needs to be checked: 1) if block after $h$ (the one in which the registration happened) is available, 2) that the height is not greater than $h+B$ with $B$ being the validity period in blocks. -The original node is removed only after the donor successfully completes reassignment, ensuring that the invariant is never violated. +[^4]: Even if on a POA chain, and no randao, this seed cannot be known to the registering provider and their colluding associates, but nonetheless should be deterministic once its set. ---- +#### Neighbourhood -## Depth Reduction +This component must be available as a public readonly endpoint taking a node's ($j\text{-th}$) ether address ($a^\Xi_j$) as a single argument. -If $N = 2^{d-1}$, every pair contains exactly one node and no donor exists. In this case, the representation is reduced by setting $d \leftarrow d-1$ and mapping each pair to a single index. This transformation is deterministic because no pair contains more than one node. +A random nonce $\varrho_n$ is used to select a neighbourhood $nh$ for a provider from the remaining unassigned neighbourhood list of level $d$: ---- +$$ +i:=\varrho\mod len(R_d) +$$ -## Interface +$$ +nh=R[i] +$$ -| Function | Description | -|----------|------------| -| Register(a) | Records commitment and blockheight | -| GetPrefix(a) | Returns assigned index | -| Validate(a,o) | Verifies overlay matches assigned index | -| Insert(a,o) | Inserts node and updates structure | -| Deregister(a) | Initiates removal | -| Remove(i) | Removes node from index | -| Rebalance(i) | Repairs invariant | +#### Checking the overlay ---- +The overlay (obtained by mining the nonce) is checked to fall in the correct neighbourhood r: +The check validates the address $a^O_n$ if and only if: -## Sequence Diagram - -```mermaid -sequenceDiagram - participant Client - participant Registry - participant Trie - - Client->>Registry: Register(a) - Registry-->>Client: prefix - Client->>Registry: Validate(a,o) - Registry->>Trie: update(F,E) - Client->>Registry: Deregister(a) - Registry->>Trie: check invariant - Registry->>Trie: select donor - Registry->>Trie: update -``` +$$ +r=a^O_n\gg(255-d) +$$ ---- +#### Assignment -## Economic Considerations +If the overlay check passes, -Fairness of the system requires that each node is associated with a fixed stake. Under this assumption, total stake is proportional to the number of nodes, +- the nodes' overlay address is assigned to a neighbourhood of depth $d$. $$ -\text{Total stake} \propto N, +A_d[r]=a^O_n $$ -and no operator gains advantage by concentrating stake into fewer nodes. This aligns economic incentives with the structural invariant. +- $N$ is incremented +- the $i$-th item is removed from remaining open neighbourhood list $R_d$. +- if $R_d$ is now of zero length, the $d$ is incremented, and new assignment and remaining lists are initialised as per section 'initialisation' above. +- provider's entry is removed from the committers' list. ---- +### Economic Analysis -## Impact -The system achieves uniform distribution of nodes, resistance to adversarial placement, and predictable operational costs. Rebalancing introduces a dependency on donor behaviour; however, the resulting delay is bounded and practically absorbed by the off-boarding process, which already operates as a queue. Consequently, rebalance latency does not introduce observable instability. ---- +### Further endpoints -## Security Analysis +A public read-only endpoint exists for querying neighbourhoods as well as nodes. Accessor for $d$ and $N$ will return the current neighbourhood depth and the current number of assigned neighbourhoods. A public accessor for $A_d$ will return for a neighbourhood (between $0$ and $2^d-1 inclusive) the overlay of the node assigned to that neighbourhood. Another endpoint will return for any overlay $o$ the closest node, so that the network service can find responsible nodes for any task with address in the space shared by overlays: -The commit-and-delay entropy mechanism prevents nodes from predicting or influencing their placement. Structural constraints prevent clustering, making Sybil attacks costly and ineffective. Validator influence is limited to a single block and does not provide sufficient control to bias assignment. - -Rebalancing cannot be blocked as long as $N > 2^{d-1}$, since a donor must exist. If a donor fails to complete reassignment, expiry ensures that another donor is selected, guaranteeing progress. - ---- +$$ +g(a)=A_d[a\gg(255-d)] +$$ -## Testing +### Deregistration -Correctness requires verifying that the invariant is preserved under all state transitions. Simulation should confirm that assignment remains uniform, that rebalancing converges, and that depth transitions occur at the correct thresholds. +Only called from the Staking contract, deregister deletes the entry for the neighbourhood belonging to the given address, makes the neighbourhood available in $R$ ---- -## Conclusion +## Implementation notes + +### Changes to the staking contract + +### Changes to the bee client + +A new endpoint to bee client must be added to register a node that is not yet registered to be assigned a neighbourhood. Once the neighbourhood is known, the client can mine the nonce needed to place the overlay in the required neighbourhood. + +### Migration + +Since a new updated staking contract, a stake migration will be needed for the upgrade. Before the change, all the simplification of the staking contract is recommended, especially to allow fixed stake in order to realign redundancy +of storage and monetary incentive: with a fixed amount staked, total stake is linearly proportional to the number of nodes, and therefore comparisons across neighbourhoods can be made based on the number of nodes. In particular, the arbitrary balanced assignment makes sense in terms of incentives (expected revenue). + +### Putting a node in each neighbourhood. + +## Contract + +```sol +// SPDX-License-Identifier: MIT +pragma solidity ^0.8.19; + +contract BalancedNeighbourhoodRegistry { + // -------------------- + // Configurable values + // -------------------- + uint256 public constant STAKE = 10,000,000,000,000,000 wei; // Stake required + uint256 public constant VALID_FOR = 128; // Validity window in blocks + + // -------------------- + // Structs and Storage + // -------------------- + struct Entry { + address committer; + uint256 height; + } + + Entry[] public committers; // List of committers (registered but as yet unassigned nodes) + mapping(address => bool) public hasCommitted; // Track if an address has committed + + uint256 public N; // Number of assigned nodes + uint256 public d = 1; // Neighbourhood depth + uint256 public currentPower = 2; // 2^d + + // Assignments: overlay address for neighbourhoods of depth d + bytes32[] public A; + + // Remaining unassigned neighbourhoods of depth d + uint256[] public R; + + // -------------------- + // Events + // -------------------- + event Registered(address indexed node, uint256 blockHeight); + event Assigned(address indexed node, uint256 neighbourhood, bytes32 overlay); + event DepthUpgraded(uint256 newDepth); + event Unassigned(address indexed node, uint256 neighbourhood); + + // -------------------- + // Registration endpoint + // -------------------- + function register() external payable { + require(msg.value == STAKE, "Invalid stake"); + require(!hasCommitted[msg.sender], "Already registered"); + + committers.push(Entry(msg.sender, block.number)); + hasCommitted[msg.sender] = true; + + emit Registered(msg.sender, block.number); + } + + // -------------------- + // Expire old entries + // -------------------- + function _expire() internal { + while (committers.length > 0) { + Entry storage e = committers[0]; + if (block.number <= e.height + VALID_FOR) { + break; + } + hasCommitted[e.committer] = false; + _removeCommitter(0); + // Burn logic: funds stay locked. + } + } + + // -------------------- + // Remove committer from the list + // -------------------- + // This function is used internally to remove a committer from the list. + // It shifts the elements to the left and pops the last element. + function _removeCommitter(uint index) internal { + if (index >= committers.length) return; + + for (uint i = index; i < committers.length - 1; i++) { + committers[i] = committers[i + 1]; + } + committers.pop(); + } + + // -------------------- + // Find entry for a committer in the committer list + // Returns the index of the entry if found, otherwise reverts. + // -------------------- + function _findEntryFor(address _a) internal view returns (uint) { + for (uint i = 0; i < committers.length; i++) { + if (committers[i].committer == _a) { + return i; + } + } + revert("Not registered"); + } + + // -------------------- + // Randomness + // -------------------- + function _randomSeed(address _a) internal view returns (uint256) { + uint i = _findEntryFor(_a); + uint256 h = committers[i].height; + // Ensure the block number is valid as expire may not have been called + require(block.number > h + 1, "Too early"); + require(block.number <= h + VALID_FOR, "Registration expired"); + // Use blockhash to generate a random seed + bytes32 bh = blockhash(h + 1); + return uint256(keccak256(abi.encodePacked(bh, _a))); + } + + // -------------------- + // Public View: Neighbourhood + // -------------------- + function getNeighbourhood(address _a) public view returns (uint256) { + uint256 r = _randomSeed(_a); + require(R.length > 0, "No available neighbourhoods"); + return R[r % R.length]; + } + + // -------------------- + // Assign node overlay to neighbourhood + // -------------------- + function assign(address _a, bytes32 _overlay) external { + _expire(); + + // Check registration + uint256 nh = getNeighbourhood(_a); + uint256 overlayNh = uint256(_overlay) >> (256 - d); + + require(overlayNh == nh, "Overlay doesn't match neighbourhood"); + + A[nh] = _overlay; + N++; + + // Remove neighbourhood from R + _removeFromR(nh); + + // Check if R is empty + if (R.length == 0) { + _upgradeDepth(); + } + + hasCommitted[_a] = false; + _removeEntry(_a); + + emit Assigned(_a, nh, _overlay); + } + + //---------------------- + // unregister + //---------------------- + function unregister(address _a, bytes32 _overlay) external { + uint nh = uint256(_overlay) >> (256 - d); + A[nh] = bytes32(0); + R.push(nh); + N--; + // return funds to the committer + payable(_a).transfer(STAKE); + emit Unassigned(_a, nh); + } + + // -------------------- + // Internal functions to manage R and committers + // -------------------- + function _removeFromR(uint256 nh) internal { + for (uint i = 0; i < R.length; i++) { + if (R[i] == nh) { + // replace the removed element with the last element and pop + R[i] = R[R.length - 1]; + R.pop(); + return; + } + } + } + + function _removeEntry(address _a) internal { + uint i = _findEntryFor(_a); + require(i < committers.length, "Entry not found"); + _removeCommitter(i); + } + + // -------------------- + // Expand A and R when needed + // -------------------- + function _upgradeDepth() internal { + currentPower = 2 ** d; + + delete R; + for (uint i = 0; i < currentPower; i++) { + R.push(0); + // Ensure A has enough space for the new neighbourhoods); + A.push(bytes32(0)); + } + for (uint i = currentPower - 1; i < currentPower; i--) { + uint b = uint256(A[i]) >> (256 - d) % 2; + A[2*i+b] = A[i]; + uint256 j = 2*i+1-b; + if (2 * j < currentPower) { + A[j] = bytes32(0); // Clear the old address + } + R[i] = j; + } + d++; + require(d <= 32, "Maximum depth exceeded"); + emit DepthUpgraded(d); + } + + // -------------------- + // Accessors + // -------------------- + function getDepth() external view returns (uint) { + return d; + } + + function getN() external view returns (uint256) { + return N; + } + + function getOverlayForNeighbourhood(uint256 nh) external view returns (bytes32) { + return A[nh]; + } + + function getClosestNode(bytes32 key) external view returns (bytes32) { + uint256 prefix = uint256(key) >> (256 - d); + return A[prefix]; + } +} +``` -This proposal replaces global restructuring with local invariant enforcement over a compact data structure. It achieves fairness, efficiency, and robustness through deterministic rules and probabilistic assignment, making it suitable for large-scale decentralised operation. From 714e14fc12586b3b9337e75bbfa89755d52f75ae Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Fri, 12 Jun 2026 14:10:21 +0200 Subject: [PATCH 15/23] Revise SWIP-39 for clarity on neighbourhood management Refactor SWIP-39 to clarify balanced neighbourhood registry and address assignment process. --- SWIPs/swip-39.md | 632 ++++++++++++++++++++++------------------------- 1 file changed, 302 insertions(+), 330 deletions(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index d48749f1..e5ef6940 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -13,201 +13,403 @@ created: 2025-07-21 ## Abstract -This SWIP introduces a systematic way for operators to enter the network in such a way that their overlay addresses are balanced within the address space. Importantly, operators must register their commitment to take part and on doing so will be assigned a random neighbourhood from the queue of those next to be occupied. Since a joining node operator is unable to position themselves at a specific point in the network without a significant time or financial penalty, attack vectors that rely on this are rendered infeasible. +This SWIP introduces a systematic way for node operators to enter the Swarm network in such a way that they form a _balanced subnetwork_. +In the context of this SWIP, _balance_ means that the distribution of nodes participating in the subnetwork be as dispersed as possible across the Swarm address space. ## Motivation -There are multiple considerations that motivate such a scheme: +### Balance and area of responsibility -- **load-balancing**: a decentralised service network will be fair if tasks are distributed to nodes so that the workload assigned to each participant is roughly equal. Presuming load balancing is achieved if tasks[^1] are assigned based on uniform random label (i.e., the content hash of the descriptor) and nodes providing the service are balanced in the address space. +The most obvious use case for a balanced sub-network is a _decentralised service network_ _(DSN)_, a set of nodes that commit to collectively perform some task. Instances of this task submitted by the users of the DSN +are best thought of as a partially ordered set of _input/output jobs_. These jobs are then assigned to the service nodes in the DSN based on whether the _job ID_ falls within the node's _area of responsibility_. Execution is load balanced, as long as: -[^1]: it is assumed that average resource utilisation (network/computation/storage requirements) of tasks over a typical period of payment remains within a tolerable variance. +- jobs are of comparable complexity, +- job IDs are random and uniform within the address space (the hash of their description), and +- nodes' areas of responsibility are address ranges of equal size. -- **arbitrary neighbourhood assigment**: the system needs to make sure that assignment of an overlay address to participant nodes is arbitrary. In particular, it is impractical (expensive) for any operator to attempt to place several nodes in the same storage neighbourhood without truely replicating storage and yet get paid. Note that, this scheme constitutes an effective solution to the problem of "one operator, one node in a neighbourhood". Taking the storage incentive system as an example, this will mitigate this sybil attack, without resorting to the rather weak incentive of additive stake as a proof of redundancy.[^2] +_Areas of responsibility_ are defined by _proximity_, ie., a contiguous range of addresses close to each other and to the node's ID (i.e., the node overlay address is in the same address space) using logarithmic distance as a metric. -[^2]: the idea is that if stake is variable and earnings are linearly proportional to earnings then, mutatis mutandis, it is always more economical for one operator to run just one node with all the stake than several nodes due to the added operational costs. ↩ +The design achieves -- **extensible and user friendly approach**: the current approach taken by swarm is based on the idea is that, if stake is variable and earnings are linearly proportional to earnings then, mutatis mutandis, it is always more economical for one operator to run just one node with all the stake than several nodes due to the added operational costs. While this has proven to be an effective means to secure the protocol, it has also led to some confusion and it's complexity means it is difficult to develop. The proposed system is intended to be much simpler to reason with and for operators to use and understand. +- fairness, +- bounded cost of operation, and +- resistance to manipulation. -## Architecture +### Further support when applied to the current postage redistribution game -The network $S$, comprises the set of all network provider nodes $\{a_0, \ldots, a_n\}$, with cardinality N. +#### Sybil attacks -Each node $a$ has an associated ether address $a_i^\Xi$ and will determine a 32 byte overlay address $a_i^\theta$. +The neighbourhood sybil attack is when the same operator runs several nodes (or runs one client node, but plays with several) in the same neighbourhood. This would allow them to share storage without replication and yet get paid. +To mitigate this we resort to the rather weak incentive of additive stake as a proof of redundancy. If stake is variable and is linearly proportional to earnings, then, mutatis mutandis, due to the added operational costs, it is always more economical for one operator in a neihgbourhood to run just one node with all the stake than several nodes. +Random NH assignnment makes it impractical (expensive) for any operator to attempt to place several nodes in the same storage neighbourhood The proposed scheme solves the problem of "one operator, one node in a neighbourhood". -$ -N = |S|, \quad S = \{a_0, \ldots, a_n\} \quad a_i^{\theta} \in \mathbb{O}_{32}\,,\ a_i^\Xi \in \mathbb{E} -$ -Then the current depth of the node assignment tree is defined as: +#### Fixed stake -$ -d_c = \lfloor \log_2(N) \rfloor + 1, \quad d_c \in \mathbb{N} -$ +Variable stake is not really compatible with random assignment. If a candidate node is assigned a neighbourhood with high stake density, it can earn less with the same stake, which is not really fair. Fixed stake across neighbourhoods, on the other hand, does not imply any a priori (dis)advantage. Uniform prices could and should allow changes over time. -The balanced node assignments are orchestrated by a smart contract which will maintain the active node list, their corresponding neighbourhoods, the overlay addresses they have determined, and manage the ingress/egress processes as nodes joini and exit the set of active network service providers. +#### Shadow world fabrication attack -This contract is deployed together with a staking contract similar to the [swarm storage incentive staking contract](https://github.com/ethersphere/storage-incentives/blob/master/src/Staking.sol). This contract will retain the total stake treasury, as well as enabling a node operator to deposit, withdrawal and maintain their stake. Concerns should be strictly separated to improve security of locked funds and upgradability of both contracts. +In order to control the stamp at game time, attackers must invest the same amount of stamp resources as the entire swarm's used capacity. Assuming that the average utilisation rate over a relavant period is $0d$. +$$ +NH(p,d)=\left\{a\in\mathbb{\Sigma}^{256} \,\mid\,a[0:d]=p[0:d]\right\} +$$ +Given a set of nodes $S$, a node $n_i\in S$ is _unique at depth_ $u_i$ if $u_i$ is the smallest integer such that no other node fall in its neighbourhood (designated by its overlay $o_i$ at depth $u_i$): +$$ +\forall 0\leq j +This entails that the depth that all nodes are unique at are equal or just has 1 as a difference: +$$ +\forall 0\leq i, j ,$) to the end of committers list. +The node assignment contract is composed of several transactional endpoints: -#### Validity -The entry is valid for a period of $B, \quad B < 256 $ blocks after the registration. +### Registering and Random Assignment -$B$ must be less than $256$, the number of blocks for which the blockhash is available from within the EVM. ((unless the blockhash is recorded)) +Candidate nodes end up assigned to a random free neighbourhood in a way that all the potential free neighbourhoods had the same chance of being selected. -Since the blockheight values of the list items are monotonically increasing, entries at the beginning of the list expire first. By iterating upto the first valid entry, expired entries can be iterated on efficiently. +#### Registration ->> SIG//NOTE should have time limit on commiting to assigned neighbourhood overlay before providingn nonce. this queing has some unintended consequences in terms of enabling squatting or blocking. the economic disincentives must be calculated and parameters/constants and/or slashing formula created i.e. how much to probably block one neighbourhood then subsequently provoke a split causing data loss. +In the first step, a node's intention to participate as a provider in the service network gets recorded in the _commit queue_ $C$. +The current blockheight $h_i$ is recorded together with the ether address by pushing the entry struct ($e_i = \langle a_i,h_i\rangle$) to the end of commit queue. -### Expiry +At the time of registering, we check if the node's ether address is not already on the list. +In order to prevent repeated trials, each node must be registered only once. +A non-refundable application fee ${\$_a}$ is deposited. -This function call iterates through all expired entries, burns their deposit, and, by setting the head of the list to the first valid item, removes them from the committer's list. -This is called by the assign function (itself called by the staking contract) before the read only call checking if the resulting overlay address falls into the neighbourhood that the registrant was assigned to, i.e., the correctness of the nonce submitted from the perspective of the staking contract. -### Assign -The assign call is the second transactional endpoint called by the staking contract. It takes the provider's ether address and as well as the mined overlay as arguments. -After calling expiry, the validity of the registration is checked by finding the entry for the ether address in the committers' list. +### Get prefix -#### Initialisation +### Get an overlay prefix assigned -Let $\overline{N}=2^d$ be the lowest power of 2 that is greater than $N$, the number of already assigned registrants, where $d$ and $N$ are defined as before. Let $R$ be the array of the remaining unassigned neighbourhoods of this level (i.e., $|R|=\overline{N}-N$). +This function includes a read-only call that and returns the neighbourhood that the node is currently assigned to. This call is public so that the client can enquire about the neighbourhood they are assigned to -Whenever the number of slots at this depth, $|R|$ drops to $0$, $d$ is incremented and $\overline{N}=2^d$ adjusts. At any point in time, a $uint256$ array of length $\overline{N}$ is maintained, called the *assignments list* $A$ holding the currently registered nodes' overlay addresses. +This public read only call takes as argument a node's ether address $a_i$ returns the current prefix contraing the overlay assignment. +Note that calling the function twice may result in a different constraint prefix if there is another successful assignment in between the two calls. -Whenever $d$ changes, new arrays for assigments $A_d$ and remainders $R_d$ are created (both twice the size of the previous one $A_{d-1}$ and $R_{d-1}$). We iterate through the current array and for each overlay address $a_i^\theta$ at position $i$ copy $a_i^\theta$ to position $2i +b_i$ where $b_i=1$ if the $d$-th bit of the address is set: +If the resulting overlay address falls into the neighbourhood that the registrant was assigned to, i.e., the correctness of the nonce submitted from the perspective of the staking contract. -$$ -b_i=a^\theta_i[d/8] -$$ +## Assign -$$ -b_i\gg=7-(d\mod 8) -$$ +The assign call is the second transactional endpoint called by the staking contract. It takes the provider's ether address as well as the mined overlay as arguments. + +### Expiry + + +The entry is valid for a period of $G$ blocks after the registration. In practice, $G$ must be less than $256$, the number of blocks for which the blockhash is available from within the EVM. + +Since the blockheight values of the commit queue items are monotonically increasing, entries at the beginning of the list expire first. By iterating upto the first valid entry, expired entries can be iterated on efficiently. + +The `expire` function call iterates the commit queue from the oldest, going through all expired entries, burns their deposit, and, by setting the head of the list to the first valid item, removes them from the front of the commit queue. + +After calling `expire`, the validity of the registration is checked by finding the entry for the ether address in the commit queue. +### Entropy +Nodes derive randomness from a high entropy seed $$ -b_i{\land\hspace{-3pt}=}1 +\rho_i = H(\text{blockhash}(h_i+1) \parallel h_i \parallel a_i), $$ +which is not known at the time of registration. The _validity window_ $VW < 256$ ensures that the referenced blockhash remains accessible. -In fact, $A_d$ stands for nodes of a binary tree on level $d$ and the value at each position is the overlay address filling that neighbourhood. When $d$ is incremented, we will need to fill some of the neighbourhoods with the existing nodes. Each node will fill the left child node (when $b_i=0$) or the right one ($b_i=1$) depending on the subsequent bit in their overlay. +### Mine a nonce +The commited node, upon learning the neighbourhood $\mathit{nh}_i$, find a nonce $\nu_i$ to generate the overlay address which is: $$ -\forall\ 0\leq i < 2^d,\quad A_d[2i+b_i]=A_{d-1}[i] +o_i := \mathit{H}(a_i \parallel networkID \parallel \nu_i) $$ +that falls in the correct neighbourhood. +$$ +\nu_i \leftarrow \mathbb{\Sigma}^{256}, \quad o_i\gg(255-d)=\mathit{nh}_i +$$ +The mined overlay $o_i$ must be submitted to the contract, which once the overlay is verified, removes the entry from the commit queue. -As we are filling the assignment list, we know that the whenever a neighbourhood is filled with an already existing node, its sister neighbourhood will be unassigned, therefore we can just record those in the remaining list. +Given $F(1)=2^{d+1} - N$ is the number of free neighbourhoods currently free. A node computes $$ -\forall\ 0\leq i < |R_d|,\quad R_d[i]=2i+1-b_i +k_i = \rho_i \bmod 2^{d+1}-N. $$ +The neighbourhoods nodes can be allocated to a cell $j=c_{k_i}$ only if $Free(j)$ is true. +The assigned index is determined by descending the trie. At a node index $j$, $F(Left(j))$ denotes the number of free slots in the left subtree. If $k < F(Left(i))$, the traversal continues to the left child. Otherwise, the traversal continues to the right child with updated rank $k_i \mapsto k_i - F(Left(i))$: + +```mermaid +flowchart LR + A([Start]) --> AA["$j=1, d=0, c=''$"] + AA --> BB{"$V(j)=\varnothing?$"} + BB -->|Yes| CC{"$F(j)=1?$"} + BB -->|No| DD["$w=V(j)$"] + DD --> CC + CC -->|Yes| DD["$c=c\parallel \not w[d]$"] + DD --> X[(end)] + CC -->|No| B{"$k|Yes| C["$j=Left(j)$"] + B -->|No| D["$k=k-F(Left(j))$
$j=Right(j)$"] + C --> E["$d++$"] + D --> E + E --> AA +``` + +## Deregistration and Rebalancing -#### Random seed +Nodes are free to deregister at any time. +If the sister node exists, removal proceeds directly and the invariant remains satisfied. -This internal read-only call takes as its single argument an ether address ($a^\Xi$) and returns a (?) random $\rho \in \mathbb{Z}_\text{uint256}$. -After checking if the ether address is a valid registrant by finding the corresponding first (unique) $entry$ struct,[^33] the `difficulty` (randao) is called on the block subsequent to the blockheight registered. Append to it the ether address and hash using $H$ it to yield what will serve as the random seed for this provider.[^4] +If removal would leave both child of the parent empty, then _rebalancing_ is required. A donor pair is selected using the same rank-based traversal over $F(c)$. From the selected pair, one of the two nodes is chosen and removed. The donor node is reinserted into the commit queue and assigned to the empty pair. -$$ -\sigma=\mathit{blockAtHeight}(\mathit{entry}(a^\Xi).\mathit{height}+1).\mathit{difficulty}() -$$ +The original node is removed only after the donor successfully completes reassignment, ensuring that the invariant is never violated. In order that the rebalancing cannot be manipulated, ie., the selected node reinserted into the neighbourhood of the deregistrant, the donor must to be selected with proper randomness, not known at the time of deregistration. +Given $\mathbb{F}(1)=N-2^{d}$ is the number of free neighbourhoods currently full (doubly filled). A node computes $$ -\varrho=\mathit{uint256}(H(\sigma|a^\Xi)) +k_i = \rho_i \bmod N-2^{d} $$ -[^33]: Since expiry is not necessarily called when the random seed it called, the blockheight needs to be checked: 1) if block after $h$ (the one in which the registration happened) is available, 2) that the height is not greater than $h+B$ with $B$ being the validity period in blocks. +The neighbourhoods nodes can be allocated to a cell $j=c_{k_i}$ only if $Free(j)$ is true. +The assigned index is determined by descending the trie. At a node index $j$, $F(Left(j))$ denotes the number of free slots in the left subtree. If $k < F(Left(i))$, the traversal continues to the left child. Otherwise, the traversal continues to the right child with updated rank $k_i \mapsto k_i - F(Left(i))$: -[^4]: Even if on a POA chain, and no randao, this seed cannot be known to the registering provider and their colluding associates, but nonetheless should be deterministic once its set. +## Specification + +### Registration -#### Neighbourhood +An initially empty list (_commit queue_) of _entry struct_ types holds the current committers. The struct holds information about the ether address of the node and the blockheight the address registered at. -This component must be available as a public readonly endpoint taking a node's ($j\text{-th}$) ether address ($a^\Xi_j$) as a single argument. +## Data Structure -A random nonce $\varrho_n$ is used to select a neighbourhood $nh$ for a provider from the remaining unassigned neighbourhood list of level $d$: +The assignment structure is implemented as an implicit complete binary trie over the index space. Each node $v$ of the trie corresponds to a contiguous interval of indices. The subtrie has the role of maintaining two quantities. +### Counting free neighbourhoods for candidate assignment + +The first quantity stands for the number of free slots in the subtree rooted at index $i$; these are tracking the number of candidate neighbourhoods to assign. $$ -i:=\varrho\mod len(R_d) +F: \mathbb{N}\to\mathbb{N}\\ +F(i) = \begin{cases} +F(Left(i)) + F(Right(i))&\text{if } Depth(i) 1 + 1 --> 2 + 1 --> 3 + 2 --> 4 + 2 --> 5 + 3 --> 6 + 3 --> 7 + 4 --> 8 + 4 --> 9 + 5 --> 10 + 5 --> 11 + 6 --> 12 + 6 --> 13 + 7 --> 14 + 7 --> 15 +``` -- the nodes' overlay address is assigned to a neighbourhood of depth $d$. +The implicit binary structure means the represented tree can be traversed using basic arithmetic on the indexes: $$ -A_d[r]=a^O_n +\begin{array}{l|l|l} +\mathrm{description} & \mathrm{notation} & \mathrm{definition}\\\hline +\text{parent of }i& \mathrm{Parent}(i) & i/2 &\\ +\text{left child of }i&\mathrm{Left}(i) & 2i\\ +\text{right child of }i& \mathrm{Right}(i) & 2i+1\\ +\text{sister of }i& \mathrm{Sister}(i) & \mathrm{Parent}(i\mathrm{Parent}(i)) + \mathrm{abs}(\mathrm{Right}(\mathrm{Parent}(i)))\\ +\text{depth of }i& \mathrm{Depth}(i) & \mathrm{Floor}(\log_2(i))\\ +\text{position of }i& \mathrm{Pos}(i) & i \mod \mathrm{Depth}(i) +\end{array} $$ -- $N$ is incremented -- the $i$-th item is removed from remaining open neighbourhood list $R_d$. -- if $R_d$ is now of zero length, the $d$ is incremented, and new assignment and remaining lists are initialised as per section 'initialisation' above. -- provider's entry is removed from the committers' list. - -### Economic Analysis - +When the index structure is used as a map, the rule of interitance allows you to look up a value that was 'inherited' from an earlier stage (inserted at a shallower depth). We can define $V$ as a lookup function for a map over the above index structure, then $V!$ is +$$ +V!(i)=\begin{cases} +V(\mathrm{Parent}(i)) &\text{if } V(i)=\varnothing\text{ and }i>1\\ +V(i) &\text{otherwise} +\end{cases} +$$ -### Further endpoints +We can define the predicate _not assigned_ as follows: +$$ +NA(i) \leftrightarrow V!(i) = \varnothing . +$$ +This allows us to define free and fully assigned neighbourhoods: +$$ +\mathrm{Free}(i) \leftrightarrow NA(\mathrm{Left}(i)) \lor NA(\mathrm{Right}(i)) +$$ +and +$$ +\mathrm{Full}(i) \leftrightarrow !NA(\mathrm{Left}(i)) \land !NA(\mathrm{Right}(i)) +$$ -A public read-only endpoint exists for querying neighbourhoods as well as nodes. Accessor for $d$ and $N$ will return the current neighbourhood depth and the current number of assigned neighbourhoods. A public accessor for $A_d$ will return for a neighbourhood (between $0$ and $2^d-1 inclusive) the overlay of the node assigned to that neighbourhood. Another endpoint will return for any overlay $o$ the closest node, so that the network service can find responsible nodes for any task with address in the space shared by overlays: +The data structure operations all enforce the condition $$ -g(a)=A_d[a\gg(255-d)] +\forall i, \mathrm{Depth}(i)< d\longrightarrow V(i)\neq \varnothing \lor V(i+1)\neq \varnothing $$ -### Deregistration +which ensures that every prefix of length $d-1$ contains at least one node. This invariant contrains the admissible states of the system. -Only called from the Staking contract, deregister deletes the entry for the neighbourhood belonging to the given address, makes the neighbourhood available in $R$ +## +The IBT is used to -## Implementation notes +- assign neighbourhoods for new applicants +- find candidate donors for rebalancing +- find the closest node to an address -### Changes to the staking contract +### Further endpoints + +A public read-only endpoint exists for querying neighbourhoods as well as nodes. Accessor for $d$ and $N$ will return the current neighbourhood depth and the current number of assigned neighbourhoods. A public accessor for $A_d$ will return for a neighbourhood (between $0$ and $2^d-1 inclusive) the overlay of the node assigned to that neighbourhood. Another endpoint will return for any overlay $o$ the closest node, so that the network service can find responsible nodes for (i.e., closest to) any address in the space shared by overlays: + +$$ +g(a)=O![a\gg(255-d)] +$$ ### Changes to the bee client @@ -215,238 +417,8 @@ A new endpoint to bee client must be added to register a node that is not yet re ### Migration -Since a new updated staking contract, a stake migration will be needed for the upgrade. Before the change, all the simplification of the staking contract is recommended, especially to allow fixed stake in order to realign redundancy -of storage and monetary incentive: with a fixed amount staked, total stake is linearly proportional to the number of nodes, and therefore comparisons across neighbourhoods can be made based on the number of nodes. In particular, the arbitrary balanced assignment makes sense in terms of incentives (expected revenue). +Since a new updated staking contract, a stake migration will be needed for the upgrade. Before the change, all the simplification of the staking contract is recommended, especially to allow fixed stake in order to realign redundancy of storage and monetary incentive: with a fixed amount staked, total stake is linearly proportional to the number of nodes, and therefore comparisons across neighbourhoods can be made based on the number of nodes. In particular, the random balanced assignment makes sense in terms of incentives (expected revenue). -### Putting a node in each neighbourhood. +### Putting a node in each neighbourhood ## Contract - -```sol -// SPDX-License-Identifier: MIT -pragma solidity ^0.8.19; - -contract BalancedNeighbourhoodRegistry { - // -------------------- - // Configurable values - // -------------------- - uint256 public constant STAKE = 10,000,000,000,000,000 wei; // Stake required - uint256 public constant VALID_FOR = 128; // Validity window in blocks - - // -------------------- - // Structs and Storage - // -------------------- - struct Entry { - address committer; - uint256 height; - } - - Entry[] public committers; // List of committers (registered but as yet unassigned nodes) - mapping(address => bool) public hasCommitted; // Track if an address has committed - - uint256 public N; // Number of assigned nodes - uint256 public d = 1; // Neighbourhood depth - uint256 public currentPower = 2; // 2^d - - // Assignments: overlay address for neighbourhoods of depth d - bytes32[] public A; - - // Remaining unassigned neighbourhoods of depth d - uint256[] public R; - - // -------------------- - // Events - // -------------------- - event Registered(address indexed node, uint256 blockHeight); - event Assigned(address indexed node, uint256 neighbourhood, bytes32 overlay); - event DepthUpgraded(uint256 newDepth); - event Unassigned(address indexed node, uint256 neighbourhood); - - // -------------------- - // Registration endpoint - // -------------------- - function register() external payable { - require(msg.value == STAKE, "Invalid stake"); - require(!hasCommitted[msg.sender], "Already registered"); - - committers.push(Entry(msg.sender, block.number)); - hasCommitted[msg.sender] = true; - - emit Registered(msg.sender, block.number); - } - - // -------------------- - // Expire old entries - // -------------------- - function _expire() internal { - while (committers.length > 0) { - Entry storage e = committers[0]; - if (block.number <= e.height + VALID_FOR) { - break; - } - hasCommitted[e.committer] = false; - _removeCommitter(0); - // Burn logic: funds stay locked. - } - } - - // -------------------- - // Remove committer from the list - // -------------------- - // This function is used internally to remove a committer from the list. - // It shifts the elements to the left and pops the last element. - function _removeCommitter(uint index) internal { - if (index >= committers.length) return; - - for (uint i = index; i < committers.length - 1; i++) { - committers[i] = committers[i + 1]; - } - committers.pop(); - } - - // -------------------- - // Find entry for a committer in the committer list - // Returns the index of the entry if found, otherwise reverts. - // -------------------- - function _findEntryFor(address _a) internal view returns (uint) { - for (uint i = 0; i < committers.length; i++) { - if (committers[i].committer == _a) { - return i; - } - } - revert("Not registered"); - } - - // -------------------- - // Randomness - // -------------------- - function _randomSeed(address _a) internal view returns (uint256) { - uint i = _findEntryFor(_a); - uint256 h = committers[i].height; - // Ensure the block number is valid as expire may not have been called - require(block.number > h + 1, "Too early"); - require(block.number <= h + VALID_FOR, "Registration expired"); - // Use blockhash to generate a random seed - bytes32 bh = blockhash(h + 1); - return uint256(keccak256(abi.encodePacked(bh, _a))); - } - - // -------------------- - // Public View: Neighbourhood - // -------------------- - function getNeighbourhood(address _a) public view returns (uint256) { - uint256 r = _randomSeed(_a); - require(R.length > 0, "No available neighbourhoods"); - return R[r % R.length]; - } - - // -------------------- - // Assign node overlay to neighbourhood - // -------------------- - function assign(address _a, bytes32 _overlay) external { - _expire(); - - // Check registration - uint256 nh = getNeighbourhood(_a); - uint256 overlayNh = uint256(_overlay) >> (256 - d); - - require(overlayNh == nh, "Overlay doesn't match neighbourhood"); - - A[nh] = _overlay; - N++; - - // Remove neighbourhood from R - _removeFromR(nh); - - // Check if R is empty - if (R.length == 0) { - _upgradeDepth(); - } - - hasCommitted[_a] = false; - _removeEntry(_a); - - emit Assigned(_a, nh, _overlay); - } - - //---------------------- - // unregister - //---------------------- - function unregister(address _a, bytes32 _overlay) external { - uint nh = uint256(_overlay) >> (256 - d); - A[nh] = bytes32(0); - R.push(nh); - N--; - // return funds to the committer - payable(_a).transfer(STAKE); - emit Unassigned(_a, nh); - } - - // -------------------- - // Internal functions to manage R and committers - // -------------------- - function _removeFromR(uint256 nh) internal { - for (uint i = 0; i < R.length; i++) { - if (R[i] == nh) { - // replace the removed element with the last element and pop - R[i] = R[R.length - 1]; - R.pop(); - return; - } - } - } - - function _removeEntry(address _a) internal { - uint i = _findEntryFor(_a); - require(i < committers.length, "Entry not found"); - _removeCommitter(i); - } - - // -------------------- - // Expand A and R when needed - // -------------------- - function _upgradeDepth() internal { - currentPower = 2 ** d; - - delete R; - for (uint i = 0; i < currentPower; i++) { - R.push(0); - // Ensure A has enough space for the new neighbourhoods); - A.push(bytes32(0)); - } - for (uint i = currentPower - 1; i < currentPower; i--) { - uint b = uint256(A[i]) >> (256 - d) % 2; - A[2*i+b] = A[i]; - uint256 j = 2*i+1-b; - if (2 * j < currentPower) { - A[j] = bytes32(0); // Clear the old address - } - R[i] = j; - } - d++; - require(d <= 32, "Maximum depth exceeded"); - emit DepthUpgraded(d); - } - - // -------------------- - // Accessors - // -------------------- - function getDepth() external view returns (uint) { - return d; - } - - function getN() external view returns (uint256) { - return N; - } - - function getOverlayForNeighbourhood(uint256 nh) external view returns (bytes32) { - return A[nh]; - } - - function getClosestNode(bytes32 key) external view returns (bytes32) { - uint256 prefix = uint256(key) >> (256 - d); - return A[prefix]; - } -} -``` - From 1be1fdd5110085f91d003a6ba5bfc3cadcb7ce64 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Mon, 20 Jul 2026 13:02:41 +0200 Subject: [PATCH 16/23] Refine SWIP-39 content and fix typos Clarified descriptions and corrected typos in the SWIP-39 document regarding Sybil attacks, neighbourhood assignments, and node registration processes. Enhanced explanations of data structures and their roles in maintaining balance and coverage in the network. --- SWIPs/swip-39.md | 359 +++++++++++++++++++++++++++-------------------- 1 file changed, 205 insertions(+), 154 deletions(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index e5ef6940..158453f2 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -41,7 +41,7 @@ The design achieves The neighbourhood sybil attack is when the same operator runs several nodes (or runs one client node, but plays with several) in the same neighbourhood. This would allow them to share storage without replication and yet get paid. To mitigate this we resort to the rather weak incentive of additive stake as a proof of redundancy. If stake is variable and is linearly proportional to earnings, then, mutatis mutandis, due to the added operational costs, it is always more economical for one operator in a neihgbourhood to run just one node with all the stake than several nodes. -Random NH assignnment makes it impractical (expensive) for any operator to attempt to place several nodes in the same storage neighbourhood The proposed scheme solves the problem of "one operator, one node in a neighbourhood". +Random NH assignnment makes it impractical (expensive) for any operator to attempt to place several nodes in the same particular storage neighbourhood. The proposed scheme solves the problem of "one operator, one node in a neighbourhood" unless the operator globally owns the majority of the nodes. #### Fixed stake @@ -58,9 +58,9 @@ _Address ranges (neighbourhoods)_ are defined by a shared prefix in the binary r Randomness is derived from on-chain entropy after registration, so assignment is unpredictable at commit time and reproducible at validation time. The assigment is done by constraining the overlay address to have the initial $d$ specific bits. -The choice of constraint is donen so that the system continuously enforces _balanced coverage_. -The exact node ID is determined outside the protocol, -Using the entropy of an arbitrary nonce, then, candidate nodes are able to find (mine) a suitable overlay address that satisfies the constraint. +The constraint is chosen so that the system continuously enforces _balanced coverage_. +The exact node ID is determined outside the protocol: +using the entropy of an arbitrary nonce, then, candidate nodes are able to find (mine) a suitable overlay address that satisfies the constraint. When nodes want to leave the network, rebalancing may be necessary. @@ -71,7 +71,7 @@ Let the set of active nodes be denoted by: $$ S = \{n_0, n_1, \ldots\}, \quad N = |S|. $$ -Each node is identified by an Ethereum address $a_i \in \mathbb{\Sigma}^{160}$ and an overlay address $o_i \in \mathbb{\Sigma}^{256}$, where $\mathbb{\Sigma}=\{0,1\}$. +Each node is identified by an *Ethereum address* $a_i \in \mathbb{\Sigma}^{160}$ and an *overlay address* $o_i \in \mathbb{\Sigma}^{256}$, where $\mathbb{\Sigma}=\{0,1\}$. A _neighbourhood_ (designated by pivot address $p$ and depth $d$) is an address range characterised by sharing bit prefix with $p$ with length $>d$. $$ @@ -89,56 +89,75 @@ $$ \end{cases} $$ Now we can show that -\begin{enumerate} -\item $d=\lfloor \text{log}_2(N) \rfloor$, and -\item $D=\text{log}_2(N)$, and -\item $\mid\{ n_i\in S \mid u_i = d+1 \}\mid = 2(N-2^d)$. -\end{enumerate} +$$ +\begin{align} +\circ\ & d=\lfloor \text{log}_2(N) \rfloor\text{, and} \\ +\circ\ & \mid\{ n_i\in S \mid u_i = d+1 \}\mid = 2(N-2^d). +\end{align} +$$ -Let us define a node's _unique neighbourhood_ wrt.~S as the neighbourhood designated by the node's overlay address $o_i$ at their unique depth $u_i$: +Let us define a node's _unique neighbourhood_ wrt.~$S$ as the neighbourhood designated by the node's overlay address $o_i$ at their unique depth $u_i$: $$ -NH_{i,S} := o_i, u_i +\mathit{nh}_{i,S} := NH(o_i, u_i) $$ -The address space is fully partitioned by the nodes in S at all times, each address falls within a subnetwork node's unique disjoint neighbourhood. +When S is balanced, the address space is fully partitioned by the nodes in S at all times: *each address falls within a subnetwork node's unique disjoint neighbourhood.* $$ -\forall a\in\mathbb{\Sigma}^{256}, a\in NH_i \text{, for some }0\leq i AA["$j=1, d=0, c=''$"] - AA --> BB{"$V(j)=\varnothing?$"} - BB -->|Yes| CC{"$F(j)=1?$"} - BB -->|No| DD["$w=V(j)$"] - DD --> CC - CC -->|Yes| DD["$c=c\parallel \not w[d]$"] - DD --> X[(end)] - CC -->|No| B{"$k|Yes| C["$j=Left(j)$"] - B -->|No| D["$k=k-F(Left(j))$
$j=Right(j)$"] - C --> E["$d++$"] - D --> E - E --> AA -``` +The assign call is the second transactional endpoint called by the staking contract. It takes the provider's ether address as well as the mined overlay as arguments. +The mined overlay $o_i$ must be submitted to the contract, which, once the overlay is verified, removes the entry from the commit queue and then records the complete overlay $o_i$ associated to the node's ether address $a_i$ (it does make a difference what data structure is used, see section X.X) ## Deregistration and Rebalancing Nodes are free to deregister at any time. If the sister node exists, removal proceeds directly and the invariant remains satisfied. -If removal would leave both child of the parent empty, then _rebalancing_ is required. A donor pair is selected using the same rank-based traversal over $F(c)$. From the selected pair, one of the two nodes is chosen and removed. The donor node is reinserted into the commit queue and assigned to the empty pair. +If removal would leave both child of the parent empty, then _rebalancing_ is required: one *donor pair* of nodes is selected from among the non-free neighbourhoods of depth $d$. From the selected pair, one of the two nodes is chosen and removed. The donor node is reinserted into the commit queue and gets assigned the sister neighbourhood of the node to be removed. -The original node is removed only after the donor successfully completes reassignment, ensuring that the invariant is never violated. In order that the rebalancing cannot be manipulated, ie., the selected node reinserted into the neighbourhood of the deregistrant, the donor must to be selected with proper randomness, not known at the time of deregistration. +The original node is removed only after the donor successfully completes reassignment, ensuring that the invariant is never violated. In order that the rebalancing cannot be manipulated, ie., the selected node reinserted into the neighbourhood of the deregistrant, the donor must be selected with proper randomness, not known at the time of deregistration. This randomness is derived the same way as when we add a node: deregistration call just records the blockheight $h_i$, and the following blockhash serves as the high entropy seed for randomness: +$$ +\rho_i = H(\text{blockhash}(h_i+1) \parallel h_i \parallel a_i), +$$ -Given $\mathbb{F}(1)=N-2^{d}$ is the number of free neighbourhoods currently full (doubly filled). A node computes +Given the number of free neighbourhoods currently full (doubly filled) is $N-2^{d}$. A node computes $$ -k_i = \rho_i \bmod N-2^{d} +r_i = \rho_i \bmod N-2^{d} $$ -The neighbourhoods nodes can be allocated to a cell $j=c_{k_i}$ only if $Free(j)$ is true. -The assigned index is determined by descending the trie. At a node index $j$, $F(Left(j))$ denotes the number of free slots in the left subtree. If $k < F(Left(i))$, the traversal continues to the left child. Otherwise, the traversal continues to the right child with updated rank $k_i \mapsto k_i - F(Left(i))$: +Let $f_0, f_1, \ldots f_{N-2^{d}-1}$ be the sorted list of indexes for doubly-filled (non-free) neighbourhoods, i.e., those whose continuations are both unique at $d+1$. ## Specification ### Registration -An initially empty list (_commit queue_) of _entry struct_ types holds the current committers. The struct holds information about the ether address of the node and the blockheight the address registered at. +## Data Structures -## Data Structure +For registration and deregistration the contract maintains two distinct commit queues $C_R$ and $C_D$. +This initially empty list of _entry_ struct types holds information about the node that committed to enter or leave the DSN. The entry struct holds information about the ether address of the node and the blockheight the address registered at. -The assignment structure is implemented as an implicit complete binary trie over the index space. Each node $v$ of the trie corresponds to a contiguous interval of indices. The subtrie has the role of maintaining two quantities. +The queues are always expanded by the `(de)register` functions (tx-s) and cleaned by `expire` (called by `add/remove`). -### Counting free neighbourhoods for candidate assignment - -The first quantity stands for the number of free slots in the subtree rooted at index $i$; these are tracking the number of candidate neighbourhoods to assign. $$ -F: \mathbb{N}\to\mathbb{N}\\ -F(i) = \begin{cases} -F(Left(i)) + F(Right(i))&\text{if } Depth(i) CC + CC -->|Yes| DD["$c=c\parallel \not w[d]$"] + DD --> X[(end)] + CC -->|No| B{"$k|Yes| C["$j=Left(j)$"] + B -->|No| D["$k=k-F_0(Left(j))$
$j=Right(j)$"] + C --> E["$d++$"] + D --> E + E --> AA +``` + + ## The IBT is used to @@ -402,8 +452,7 @@ The IBT is used to - assign neighbourhoods for new applicants - find candidate donors for rebalancing - find the closest node to an address - -### Further endpoints +W### Further endpoints A public read-only endpoint exists for querying neighbourhoods as well as nodes. Accessor for $d$ and $N$ will return the current neighbourhood depth and the current number of assigned neighbourhoods. A public accessor for $A_d$ will return for a neighbourhood (between $0$ and $2^d-1 inclusive) the overlay of the node assigned to that neighbourhood. Another endpoint will return for any overlay $o$ the closest node, so that the network service can find responsible nodes for (i.e., closest to) any address in the space shared by overlays: @@ -411,6 +460,8 @@ $$ g(a)=O![a\gg(255-d)] $$ +If the resulting overlay address falls into the neighbourhood that the registrant was assigned to, i.e., the correctness of the nonce submitted from the perspective of the staking contract. + ### Changes to the bee client A new endpoint to bee client must be added to register a node that is not yet registered to be assigned a neighbourhood. Once the neighbourhood is known, the client can mine the nonce needed to place the overlay in the required neighbourhood. From 9061416a7ad2ac9fcd8d3983bede9789e2039c86 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Sun, 26 Jul 2026 07:28:54 +0200 Subject: [PATCH 17/23] Revise SWIP-39 for clarity and precision Refine language and clarify concepts in SWIP-39, focusing on balanced neighbourhood registry and node assignment processes. --- SWIPs/swip-39.md | 282 +++++++++++++++++++++++++++-------------------- 1 file changed, 163 insertions(+), 119 deletions(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index 158453f2..7ba9ff0b 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -9,12 +9,13 @@ category: Core created: 2025-07-21 --- -# Balanced Neighbourhood Registry aka Smart Neighbourhood Management - ## Abstract -This SWIP introduces a systematic way for node operators to enter the Swarm network in such a way that they form a _balanced subnetwork_. -In the context of this SWIP, _balance_ means that the distribution of nodes participating in the subnetwork be as dispersed as possible across the Swarm address space. +This SWIP specifies an on-chain registry that assigns participating Swarm nodes to overlay-address prefixes while maintaining balanced coverage of the address space. For $N$ active nodes, every node is assigned an area of responsibility at one of two adjacent depths, so the largest area is at most twice the size of the smallest. + +A commit–finalize procedure selects an available prefix using post-commit blockchain entropy. The node then mines an overlay address satisfying that prefix and activates it through the registry. When a departure would violate the balance invariant, the registry coordinates reassignment of an active node before completing the departure. + +The mechanism is intended to make targeted concentration in a particular neighbourhood probabilistic and costly, and to support balanced allocation of storage-incentive or decentralised-service workloads. It does not by itself establish operator identity or prevent one operator from registering multiple nodes. ## Motivation @@ -27,7 +28,7 @@ are best thought of as a partially ordered set of _input/output jobs_. These job - job IDs are random and uniform within the address space (the hash of their description), and - nodes' areas of responsibility are address ranges of equal size. -_Areas of responsibility_ are defined by _proximity_, ie., a contiguous range of addresses close to each other and to the node's ID (i.e., the node overlay address is in the same address space) using logarithmic distance as a metric. +_Areas of responsibility_ are defined by _proximity_, i.e., a contiguous range of addresses close to each other and to the node's ID (i.e., the node overlay address is in the same address space) using logarithmic distance as a metric. The design achieves @@ -40,9 +41,9 @@ The design achieves #### Sybil attacks The neighbourhood sybil attack is when the same operator runs several nodes (or runs one client node, but plays with several) in the same neighbourhood. This would allow them to share storage without replication and yet get paid. -To mitigate this we resort to the rather weak incentive of additive stake as a proof of redundancy. If stake is variable and is linearly proportional to earnings, then, mutatis mutandis, due to the added operational costs, it is always more economical for one operator in a neihgbourhood to run just one node with all the stake than several nodes. -Random NH assignnment makes it impractical (expensive) for any operator to attempt to place several nodes in the same particular storage neighbourhood. The proposed scheme solves the problem of "one operator, one node in a neighbourhood" unless the operator globally owns the majority of the nodes. +To mitigate this we resort to the rather weak incentive of additive stake as a proof of redundancy. If stake is variable and is linearly proportional to earnings, then, mutatis mutandis, due to the added operational costs, it is always more economical for one operator in a neighbourhood to run just one node with all the stake than several nodes adding up to the same. +Random assignment makes targeted concentration in a chosen neighbourhood probabilistic and economically costly. It does not prevent one operator from registering multiple nodes. Its effectiveness therefore depends on the per-registration stake, the non-refundable fee, penalties for abandoned assignments, and the number of identities an attacker can fund. #### Fixed stake @@ -50,14 +51,14 @@ Variable stake is not really compatible with random assignment. If a candidate n #### Shadow world fabrication attack -In order to control the stamp at game time, attackers must invest the same amount of stamp resources as the entire swarm's used capacity. Assuming that the average utilisation rate over a relavant period is $0d$. +A _neighbourhood_ (designated by pivot address $p$ and depth $d$) is an address range characterised by sharing bit prefix of length $>d$ with $p$. $$ -NH(p,d)=\left\{a\in\mathbb{\Sigma}^{256} \,\mid\,a[0:d]=p[0:d]\right\} +\newcommand{\idx}[1]{\texttt{[}\,#1\,\texttt{]}} +\mathrm{NH}(p,d)=\left\{a\in\mathbb{\Sigma}^{256} \,\mid\,\forall 0\leq i1\\ +V!(\mathrm{Parent}(i)) &\text{if } V(i)=\varnothing\text{ and }i>1\\ V(i) &\text{otherwise} \end{cases} $$ We can define the predicate _not assigned_ ($\mathit{NA}$) as follows: $$ -\mathrm{NA}(i) \leftrightarrow V!(i) = \varnothing . +\mathrm{NA}(i) \Longleftrightarrow V(i) = \varnothing . $$ This allows us to define free and fully assigned neighbourhoods: $$ -\mathrm{Free}(i) \leftrightarrow \mathrm{NA}(\mathrm{Left}(i)) \lor \mathrm{NA}(\mathrm{Right}(i)) +\mathrm{Free}(i) \Longleftrightarrow \mathrm{NA}(\mathrm{Left}(i)) \lor \mathrm{NA}(\mathrm{Right}(i)) $$ and $$ -\mathrm{Full}(i) \leftrightarrow !\mathrm{NA}(\mathrm{Left}(i)) \land !\mathrm{NA}(\mathrm{Right}(i)) +\mathrm{Full}(i) \Longleftrightarrow !\mathrm{NA}(\mathrm{Left}(i)) \land !\mathrm{NA}(\mathrm{Right}(i)) $$ The data structure operations all enforce the condition @@ -355,12 +376,17 @@ $$ \forall i, \mathrm{Depth}(i)< d\Rightarrow V(i)\neq \varnothing \lor V(i+1)\neq \varnothing $$ -which ensures that every prefix of length $d-1$ contains at least one node. This invariant contrains the admissible states of the system. +which ensures that every prefix of length $d-1$ contains at least one node. This invariant constrains the admissible states of the system. +The ICBT is used to + +- find candidate neighbourhoods for new applicants +- find candidate donors for rebalancing +- find the closest node to an address (prefix) ### Counting free neighbourhoods for candidate assignment -Function $F_0(i)$ on the index tracks the number of free slots (candidate neighbourhoods to assign) in the subtree rooted at index $i$: +Function $F_0(i)$ on the index tracks the number of free slots (candidate neighbourhoods to assign) in the left subtree rooted at index $i$: $$ F_0: \mathbb{N}\to\mathbb{N}\\ F_0(i) = \begin{cases} @@ -375,22 +401,43 @@ $$ $$ In this case, $$ -\forall 0< i<2^{D}, \quad F_0(i) = 2^{D-Depth(i)}. +\forall 0< i<2^{D-1}, \quad F_0(i) = 2^{D-Depth(i)}. $$ By the time the next depth is reached, $N=2^{D+1}-1$-th element is assigned, all of the free neighbourhoods got allocated, thus: $$ -\forall 2^{D}\leq i< 2^{D+1}, \quad F_0(i)=1 +\forall 2^{D}\leq i< 2^{D+1}, \quad F_0(i)=0 $$ -and therefore: +and therefore, $$ \forall 0< i<2^{D}, \quad F_0(i) = 0. $$ -#### Counting fully saturated leaves for donor selection +We define now a sketch of the _OverlayPref_ function. its single argument is $r_i$. + +```python +func OverlayPref( k uint32 ) returns []byte + uint32 j = 1 + uint32 c = 0 + for depth(j)<=depth() + if !NA(O(j)) o=O(j) # overlay assigned + c<<=1 # shift c + j=Left(j) # set j to left child + if k>F(j) # if k over volume on left branch + k-=F(j) # then adjust to subtree volume; + j++ # set j as the right child + c||=0x01 # right + F(j)=F(j)-1 # decrement volume + c<<=1 + if o[d+1] == 0 # must not match existing + c||=0x01 # + return rbinary.bigendian(c)[ 31-d : ] +``` + +### Counting fully saturated leaves for donor selection -The second quantity one needs to track is the number of nodes in the subtree with both their children assigned on level ${d}: these correspond to _candidate donor pairs_. -We can use F_1 +The second quantity one needs to track is the number of nodes in the subtree with both their children assigned on level ${d}$: these correspond to _candidate donor pairs_. +We can use $F_1$ $$ S: &\mathbb{N}\to\mathbb{N}\\ @@ -407,7 +454,7 @@ $$ $$ In this case, $$ -\forall 0< i<2^{D}, \quad F_1(i) = 0. +\forall 0< i<2^{D-1}, \quad F_1(i) = 0. $$ By the time $2^{D}$ nodes are assigned and the trie is again balanced having $N=2^{D+1}$ nodes, all of the free neighbourhoods got allocated, thus: @@ -419,57 +466,54 @@ $$ \forall 0< i<2^{D-1}, \quad F_1(i) = 2^{D-Depth(i)}. $$ -Surely, initially, when $N=0$, $d=0$, then $F_0(0)=1$, and $F_1(0)=0$ +Surely, initially, when $N=0$, $d=0$, then $F_0(0)=1$, and $F_1(0)=0$. Note that when $D=0, N=2^0=1$: $$ -F_0(1)+F_1(1)=2^D +F_0(i)+F_1(i)=0 \bmod 2^{d-depth()}. $$ -From the definition it is also clear that, for all other situations when $2^d CC - CC -->|Yes| DD["$c=c\parallel \not w[d]$"] - DD --> X[(end)] - CC -->|No| B{"$k|Yes| C["$j=Left(j)$"] - B -->|No| D["$k=k-F_0(Left(j))$
$j=Right(j)$"] - C --> E["$d++$"] - D --> E - E --> AA -``` +```python +func SelectDonor( k uint32 ) returns overlay + uint32 j = 1 + for depth(j)F(j) # if k over volume on left branch + k-=F(j) # then adjust to subtree volume; + j++ # set j as the right child + F(j)=F(j)-1 # decrement volume -## + if !NA(O(j)) o=O(j) # depth overlay assigned + j=Left(j) # set j to left child + j+=rand(2) # set to right with 0.5 probability + if !NA(O(j)) o=O(j) # depth overlay assigned -The IBT is used to + return o +``` -- assign neighbourhoods for new applicants -- find candidate donors for rebalancing -- find the closest node to an address -W### Further endpoints +### Further endpoints -A public read-only endpoint exists for querying neighbourhoods as well as nodes. Accessor for $d$ and $N$ will return the current neighbourhood depth and the current number of assigned neighbourhoods. A public accessor for $A_d$ will return for a neighbourhood (between $0$ and $2^d-1 inclusive) the overlay of the node assigned to that neighbourhood. Another endpoint will return for any overlay $o$ the closest node, so that the network service can find responsible nodes for (i.e., closest to) any address in the space shared by overlays: +A public read-only endpoint exists for querying neighbourhoods as well as nodes. Accessor for $d$ and $N$ will return the current neighbourhood depth and the current number of assigned neighbourhoods. A public accessor for $A_d$ will return for a neighbourhood (between $0$ and $2^d-1 inclusive) the overlay of the node assigned to that neighbourhood. Another endpoint will return for any partial address $o$ the closest node, so that the network service can find responsible nodes for (i.e., closest to) any address in the space shared by overlays: $$ -g(a)=O![a\gg(255-d)] +\begin{eqnarray} +g&:&\Sigma^{256}\to\Sigma^{256}\\ +g(a)&=&O!(a\gg(254-d)) +\end{eqnarray} $$ If the resulting overlay address falls into the neighbourhood that the registrant was assigned to, i.e., the correctness of the nonce submitted from the perspective of the staking contract. ### Changes to the bee client -A new endpoint to bee client must be added to register a node that is not yet registered to be assigned a neighbourhood. Once the neighbourhood is known, the client can mine the nonce needed to place the overlay in the required neighbourhood. +A new endpoint must be added to clients to register a node that is not yet registered to be assigned a neighbourhood. Once the neighbourhood is known, the client can mine the nonce needed to place the overlay in the required neighbourhood. ### Migration Since a new updated staking contract, a stake migration will be needed for the upgrade. Before the change, all the simplification of the staking contract is recommended, especially to allow fixed stake in order to realign redundancy of storage and monetary incentive: with a fixed amount staked, total stake is linearly proportional to the number of nodes, and therefore comparisons across neighbourhoods can be made based on the number of nodes. In particular, the random balanced assignment makes sense in terms of incentives (expected revenue). -### Putting a node in each neighbourhood - -## Contract From cd081fd88ef895ecef6350a99ddbdcef0d7af82a Mon Sep 17 00:00:00 2001 From: zelig Date: Sun, 26 Jul 2026 11:41:31 +0200 Subject: [PATCH 18/23] move swip-39.md to SWIP-39/README.md for proper math rendering Co-Authored-By: Claude Fable 5 --- SWIPs/{swip-39.md => SWIP-39/README.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename SWIPs/{swip-39.md => SWIP-39/README.md} (100%) diff --git a/SWIPs/swip-39.md b/SWIPs/SWIP-39/README.md similarity index 100% rename from SWIPs/swip-39.md rename to SWIPs/SWIP-39/README.md From d63d6a7b4260e304cc2793924cd61b63aa77b405 Mon Sep 17 00:00:00 2001 From: zelig Date: Sun, 26 Jul 2026 20:48:06 +0200 Subject: [PATCH 19/23] fix math rendering: pad display blocks with blank lines, repair LaTeX errors Move back to swip-39.md (single-file layout). Display math now sits in its own paragraphs as GitHub requires; also fixes align->aligned, missing row breaks, a stray alignment tab, an unclosed inline $, an undefined \idx macro use, and the --- SWIPs/{SWIP-39/README.md => swip-39.md} | 95 ++++++++++++++++++++++--- 1 file changed, 84 insertions(+), 11 deletions(-) rename SWIPs/{SWIP-39/README.md => swip-39.md} (96%) diff --git a/SWIPs/SWIP-39/README.md b/SWIPs/swip-39.md similarity index 96% rename from SWIPs/SWIP-39/README.md rename to SWIPs/swip-39.md index 7ba9ff0b..6571b6ea 100644 --- a/SWIPs/SWIP-39/README.md +++ b/SWIPs/swip-39.md @@ -51,7 +51,7 @@ Variable stake is not really compatible with random assignment. If a candidate n #### Shadow world fabrication attack -In order to control the stamp at game time, attackers must invest the same amount of stamp resources as the entire swarm's used capacity. Assuming that the average utilisation rate over a relevant period is $0d$ with $p$. + $$ \newcommand{\idx}[1]{\texttt{[}\,#1\,\texttt{]}} \mathrm{NH}(p,d)=\left\{a\in\mathbb{\Sigma}^{256} \,\mid\,\forall 0\leq i Date: Mon, 27 Jul 2026 01:33:43 +0200 Subject: [PATCH 20/23] no prefix reservation; hard balance via donor takeover with stake forfeiture; data handover scope note - Abstract: prefix derivation is a read-only calculation, no reservation held by the registry; a taken neighbourhood simply yields a new target - Rebalancing: donor's neighbourhood is taken over by its sister (a balanced removal), donor re-enters the commit queue with blockheight set by the deregister call, and must relocate within the validity window or forfeit its stake; a fresh donor is then drawn - New 'Data handover' subsection scoping content migration out of this SWIP, deferring to upcoming durability guarantees / cold storage Co-Authored-By: Claude Fable 5 --- SWIPs/swip-39.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index 6571b6ea..c10ad0c8 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -13,7 +13,7 @@ created: 2025-07-21 This SWIP specifies an on-chain registry that assigns participating Swarm nodes to overlay-address prefixes while maintaining balanced coverage of the address space. For $N$ active nodes, every node is assigned an area of responsibility at one of two adjacent depths, so the largest area is at most twice the size of the smallest. -A commit–finalize procedure selects an available prefix using post-commit blockchain entropy. The node then mines an overlay address satisfying that prefix and activates it through the registry. When a departure would violate the balance invariant, the registry coordinates reassignment of an active node before completing the departure. +After committing, a node derives its target prefix deterministically from post-commit blockchain entropy — a read-only calculation, with no reservation held by the registry. The node mines an overlay address satisfying the prefix and activates it through the registry; if the neighbourhood has been taken in the meantime, the calculation simply yields a new target. When a departure would violate the balance invariant, the registry reassigns an active node to the vacated position before completing the departure; the selected node must relocate or else it is dropped and its stake forfeited. The mechanism is intended to make targeted concentration in a particular neighbourhood probabilistic and costly, and to support balanced allocation of storage-incentive or decentralised-service workloads. It does not by itself establish operator identity or prevent one operator from registering multiple nodes. @@ -292,7 +292,7 @@ The mined overlay $o_i$ must be submitted to the contract, which, once the overl Nodes are free to deregister at any time. If the sister node exists, removal proceeds directly and the invariant remains satisfied. -If removal would leave both children of the parent empty, then _rebalancing_ is required: upon calling `deregister`, one _donor pair_ of nodes is selected from among the full (doubly filled) neighbourhoods of depth $d$. From the selected pair, one of the two nodes is chosen and removed. The donor node is reinserted into the commit queue and gets assigned the sister neighbourhood of the node to be removed. +If removal would leave both children of the parent empty, then _rebalancing_ is required: upon calling `deregister`, one _donor pair_ of nodes is selected from among the full (doubly filled) neighbourhoods of depth $d$. From the selected pair, one of the two nodes is chosen as _donor_. The donor's neighbourhood is taken over by its sister, so the donor is removed first exactly as if it had chosen to leave itself — balance is preserved, since the sister becomes unique at depth $d$. The donor is then entered into the commit queue as if newly registering, except that the recorded blockheight is set by the `deregister` call, and it gets assigned the sister neighbourhood of the node to be removed. The donor must complete this relocation within the validity window; otherwise it is dropped from the registry with its stake forfeited, and a fresh donor is drawn. The original node is removed only after the donor successfully completes reassignment, ensuring that the invariant is never violated. In order that the rebalancing cannot be manipulated, i.e., the selected node reinserted into the neighbourhood of the deregistrant, the donor must be selected with proper randomness, not known at the time of deregistration. This randomness is derived the same way as when we add a node: deregistration call just records the block height $h_i$, and the following blockhash serves as the high entropy seed for randomness: @@ -308,6 +308,10 @@ $$ Let $f_0, f_1, \ldots f_{N-2^{d}-1}$ be the sorted list of indexes for doubly-filled (non-free) neighbourhoods, i.e., those whose continuations are both unique at $d+1$. +#### Data handover + +This SWIP specifies only the assignment of nodes to neighbourhoods; the handover of stored content during relocation is deliberately left unspecified. A relocated node is expected to synchronise the reserve of its newly assigned neighbourhood, while the neighbourhood it vacated retains the data with the sister node. With upcoming durability guarantees, such relocation gaps become repairable events; in particular, a newly assigned node need not sync its reserve live from its sister but may instead acquire the neighbourhood's content from cold storage. + ## Specification ### Data Structures From 6506ad761f85ddf444165602afd25b3cf79503ee Mon Sep 17 00:00:00 2001 From: zelig Date: Mon, 27 Jul 2026 01:54:31 +0200 Subject: [PATCH 21/23] restructure SWIP-39: adopt full rewrite with two-tx join, no reservation Replace the document with the restructured version: scope/terminology/ non-goals, balance invariant with preservation proofs, domain-separated randomness with rejection sampling and anti-grinding measures, threat model and security analysis, gas analysis, migration plan, and worked examples with trie and traversal diagrams. Adapted to agreed decisions: join is exactly two transactions (register, activate) with the target prefix a read-only computation and no reservation held; donor relocation via sibling takeover, commit-queue re-entry at the deregister blockheight, relocate-or-forfeit with redraw; data handover explicitly out of scope, deferred to durability/cold storage. Implementation notes carry the ICBT traversal table and diagrams, targetPrefix/selectDonor pseudocode, commit-queue expiry logic, ICBT as a container contract used by the registry, and the staking-contract separation (freezing/slashing stays with staking). Co-Authored-By: Claude Fable 5 --- SWIPs/swip-39.md | 902 ++++++++++++++++++++++++++++------------------- 1 file changed, 549 insertions(+), 353 deletions(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index c10ad0c8..cd2c8e5b 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -13,584 +13,780 @@ created: 2025-07-21 This SWIP specifies an on-chain registry that assigns participating Swarm nodes to overlay-address prefixes while maintaining balanced coverage of the address space. For $N$ active nodes, every node is assigned an area of responsibility at one of two adjacent depths, so the largest area is at most twice the size of the smallest. -After committing, a node derives its target prefix deterministically from post-commit blockchain entropy — a read-only calculation, with no reservation held by the registry. The node mines an overlay address satisfying the prefix and activates it through the registry; if the neighbourhood has been taken in the meantime, the calculation simply yields a new target. When a departure would violate the balance invariant, the registry reassigns an active node to the vacated position before completing the departure; the selected node must relocate or else it is dropped and its stake forfeited. +A node joins in exactly two transactions: it first commits its identity, stake, and fee; then — after deriving its target prefix from post-commit blockchain entropy through a read-only calculation, with no reservation held by the registry — it mines a conforming overlay address and activates it. If the target neighbourhood is taken in the meantime, the calculation simply yields a new target. When a departure would violate the balance invariant, the registry reassigns an active node to the vacated position before completing the departure; the selected donor must relocate or else it is dropped and its stake forfeited. -The mechanism is intended to make targeted concentration in a particular neighbourhood probabilistic and costly, and to support balanced allocation of storage-incentive or decentralised-service workloads. It does not by itself establish operator identity or prevent one operator from registering multiple nodes. +The mechanism is intended to make targeted concentration in a particular neighbourhood probabilistic and costly, and to support balanced allocation of storage-incentive or decentralised-service workloads. It does not establish operator identity or prevent one operator from registering multiple nodes. -## Motivation +## Scope -### Balance and area of responsibility +This SWIP defines: -The most obvious use case for a balanced sub-network is a _decentralised service network_ _(DSN)_, a set of nodes that commit to collectively perform some task. Instances of this task submitted by the users of the DSN -are best thought of as a partially ordered set of _input/output jobs_. These jobs are then assigned to the service nodes in the DSN based on whether the _job ID_ falls within the node's _area of responsibility_. Execution is load balanced, as long as: +- the balance invariant maintained by the registry; +- the join, activation, and departure state transitions; +- the source and use of randomness; +- the contract state required to select prefixes and donor nodes; +- the integration points required by the swarm client; and +- the security and liveness assumptions of the mechanism. -- jobs are of comparable complexity, -- job IDs are random and uniform within the address space (the hash of their description), and -- nodes' areas of responsibility are address ranges of equal size. +The join procedure consists of two transactions, `register` and `activate`; everything between them — entropy finalization, target-prefix computation, and overlay mining — happens off-chain or through read-only calls. The registry holds no reservations: the target prefix is at all times a deterministic, read-only computation over current contract state. A target invalidated by an intervening activation simply yields a new target; under realistic churn such races are rare and cost only local mining work. -_Areas of responsibility_ are defined by _proximity_, i.e., a contiguous range of addresses close to each other and to the node's ID (i.e., the node overlay address is in the same address space) using logarithmic distance as a metric. +### Terminology -The design achieves +**Address space** — the set $\Sigma^{256}$, where $\Sigma=\{0,1\}$. -- fairness, -- bounded cost of operation, and -- resistance to manipulation. +**Overlay** — node overlay address derived from a node's public key, network ID, and overlay nonce using Bee's canonical overlay derivation function. -### Further support when applied to the current postage redistribution game +**Prefix** — a bit string $p\in\Sigma^\ell$. Its length $\ell$ is its depth. -#### Sybil attacks +**Neighbourhood** — the subset of the address space that share a prefix. -The neighbourhood sybil attack is when the same operator runs several nodes (or runs one client node, but plays with several) in the same neighbourhood. This would allow them to share storage without replication and yet get paid. -To mitigate this we resort to the rather weak incentive of additive stake as a proof of redundancy. If stake is variable and is linearly proportional to earnings, then, mutatis mutandis, due to the added operational costs, it is always more economical for one operator in a neighbourhood to run just one node with all the stake than several nodes adding up to the same. +**Leaf** — a prefix currently assigned to exactly one active node. Active leaves form a complete, prefix-free partition of the address space. -Random assignment makes targeted concentration in a chosen neighbourhood probabilistic and economically costly. It does not prevent one operator from registering multiple nodes. Its effectiveness therefore depends on the per-registration stake, the non-refundable fee, penalties for abandoned assignments, and the number of identities an attacker can fund. +**Sibling** — the other child of the same parent prefix. For $p=b_0\ldots b_{\ell-1}$, its sibling is $b_0\ldots(1-b_{\ell-1})$. -#### Fixed stake +**Split candidate** — an active leaf at the current minimum depth $d$. A join splits this leaf into two child prefixes. -Variable stake is not really compatible with random assignment. If a candidate node is assigned a neighbourhood with high stake density, it can earn less with the same stake, which is not really fair. Fixed stake across neighbourhoods, on the other hand, does not imply any a priori (dis)advantage. Uniform prices could and should allow changes over time. +**Donor pair** — two active sibling leaves at depth $d+1$. One member can move to a vacated depth-$d$ leaf while the other expands to their shared parent. -#### Shadow world fabrication attack +**Registration** — a request to join that has committed an identity, stake, and fee before its assignment entropy is known. -In order to control the stamp at game time, attackers must invest the same amount of stamp resources as the entire swarm's used capacity. Assuming that the average utilisation rate over a relevant period is $0 < u\leq 1$, the reward/cost ratio for the attacker for any wins is $r=1+\frac{1}{u}$. This implies that the attacker needs to win at least once every $r$ rounds in order just to break even. +**Deregistration** — a request to leave the sub-network. -## Solution +**Pending departure** — a departure awaiting donor relocation. -_Address ranges (neighbourhoods)_ are defined by a shared prefix in the binary representation of an address, i.e., a neighbourhood designated by $a$ of depth $d$. This SWIP describes an on-chain DSN registry, where nodes identified by their Ethereum addresses are assigned a neighbourhood at random. This involves a 2-step interaction with the blockchain; in the initial transaction, candidate nodes commit to participate by registering their address and record the blockheight. +**Active node count** — $N$, the number of activated nodes in the balanced partition. A pending registration is not active. -Randomness is derived from on-chain entropy after registration, so assignment is unpredictable at commit time and reproducible at validation time. -The assignment is done by constraining the overlay address to have the initial $d$ specific bits. -The constraint is chosen so that the system continuously enforces _balanced coverage_. -The exact node ID is determined outside the protocol: -using the entropy of an arbitrary nonce, then, candidate nodes are able to find (mine) a suitable overlay address that satisfies the constraint. +### Goals and explicit non-goals -When nodes want to leave the network, rebalancing may be necessary. +The goals of this SWIP are: -### Formal exposition +- complete coverage of the address space by disjoint areas of responsibility; +- a maximum factor of two between the largest and smallest areas; +- uniformly random selection among currently eligible split candidates or donor pairs; +- assignment that is unknown to an applicant when it commits stake and fees; +- a bounded, explicit procedure for failed joins and failed donor relocations; +- logarithmic contract work in the number of active nodes for selection and tree updates; and +- deterministic validation of assignments by the contract and Bee clients. -Let the set of active nodes be denoted by: +The following are explicit non-goals: -$$ -S = \{n_0, n_1, \ldots\}, \quad N = |S|. -$$ +- proving that two Ethereum addresses belong to different operators; +- enforcing one human or one operator per node; +- preventing a well-funded operator from registering many independently assigned nodes; +- defining the postage redistribution game itself; +- defining the stake amount, application fee, reward, or slashing schedule; and +- specifying the transfer of stored content between nodes (see [Data handover](#data-handover)). -Each node is identified by an _Ethereum address_ $a_i \in \mathbb{\Sigma}^{160}$ and an _overlay address_ $o_i \in \mathbb{\Sigma}^{256}$, where $\mathbb{\Sigma}=\{0,1\}$. +Stake, fees, activation deadlines, and relocation deadlines are nevertheless security parameters. Their values MUST be specified by deployment configuration or by the staking specification on which a deployment depends. -A _neighbourhood_ (designated by pivot address $p$ and depth $d$) is an address range characterised by sharing bit prefix of length $>d$ with $p$. +## Motivation -$$ -\newcommand{\idx}[1]{\texttt{[}\,#1\,\texttt{]}} -\mathrm{NH}(p,d)=\left\{a\in\mathbb{\Sigma}^{256} \,\mid\,\forall 0\leq i1$, all leaves are at depth $d$. Remove the departing leaf and replace its sibling with their depth-$(d-1)$ parent. +3. If $N>2^d$ and the departing leaf has depth $d+1$, remove it. +4. If $N>2^d$ and the departing leaf has depth $d$, select _a donor pair_ at depth $d+1$, move one donor to the departing prefix. + +Each transformation reduces $N$ by one and preserves the invariant once the required donor relocation has completed. + +### Join protocol + +The join protocol consists of exactly two transactions: `register` and `activate`. No reservation is held by the registry at any point. + +#### 1. Register + +The applicant submits: + +- its staking identity $a_i$; +- its Bee public key $P_i$; +- the required stake; +- a non-refundable application fee. + +The contract creates a registration with a unique request ID, records the registration block height $h_i$, and appends the entry to the registration queue. An identity MUST NOT have more than one live registration or active assignment. The stake remains locked until the registration activates or expires according to the staking rules. + +Registering does not modify the active prefix tree. + +The registration is valid for a _validity window_ of $VW$ blocks. In practice $VW < 256$, the number of blocks for which a block hash remains available from within the EVM. Since block heights in the registration queue are monotonically increasing, entries expire from the front; a permissionless `expire` call advances the queue head past expired entries and burns their fees, without ever iterating the full queue. + +#### 2. Compute the target prefix + +After the configured entropy delay has passed, the applicant's target prefix is a deterministic, read-only computation over current contract state: the contract derives a seed as specified in [Randomness and anti-grinding measures](#randomness-and-anti-grinding-measures) and selects a split candidate uniformly among those currently eligible. + +The computation is exposed as a read-only call taking the applicant's identity and returning the current constraint prefix and the registration deadline. Calling it repeatedly MAY return a different prefix if another activation landed in between; nothing is locked on the applicant's behalf. + +For $N=0$, selection is omitted and the target is the empty prefix. There is no incumbent. + +#### 3. Mine an overlay + +The applicant mines a nonce $\nu_i$ such that Bee's canonical overlay derivation: $$ -o_n\in \mathit{nh}_{k, S} +o_i=\operatorname{Overlay}(P_i,\operatorname{networkID},\nu_i) $$ -and +starts with the target prefix. + +For a prefix of length $\ell$, mining takes $2^\ell$ trials in expectation. Implementations SHOULD expose progress and the registration deadline to the operator. + +#### 4. Activate or expire + +Before the deadline, the applicant submits $\nu_i$ and any proof required to bind the call to $a_i$. The contract: + +1. derives or verifies $o_i$; +2. recomputes the target prefix from current state and verifies that $o_i$ starts with it; +3. either creates the root leaf when $N=0$, or replaces the selected leaf with the incumbent and applicant child leaves; +4. marks the applicant active; +5. updates aggregate tree counters; and +6. removes the registration from the queue. + +If the tree changed between mining and activation, the recomputed target may differ and verification fails; the applicant recomputes and re-mines within its validity window. If the window lapses without activation, the registration expires and the application fee is forfeited. Additional stake penalties, if any, are defined by the staking specification. + +### Departure and rebalancing + +Nodes are free to deregister at any time. For registration and deregistration the contract maintains two distinct commit queues, $C_R$ and $C_D$, extended by the `register`/`deregister` transactions and cleaned by `expire`. + +#### Direct collapse + +Direct collapse applies when either: + +- $N>1$ is a power of two; or +- $N$ is not a power of two and the departing leaf has depth $d+1$. + +In both cases the departing node has an active leaf sibling, and collapsing that sibling to their parent produces the depth distribution required for $N-1$. The departure completes immediately: the contract removes the departing leaf, moves the sibling to the parent, and releases the departing stake as permitted. + +If $N=1$, there is no sibling. The registry MAY remove the sole root assignment after the configured departure delay, leaving the service with no coverage. User interfaces MUST make this consequence explicit. + +#### Donor relocation + +If $N>2^d$ and the departing node is a depth-$d$ leaf, direct collapse would violate the invariant. The contract therefore relocates a donor: + +1. the departure is committed by the `deregister` call, recording its block height before donor-selection entropy is known; +2. the contract selects a donor pair uniformly and one member of the pair as the moving donor; +3. the donor's own leaf is taken over by its sibling — the donor is removed exactly as if it had chosen to leave itself, which is a balanced removal since the sibling becomes unique at depth $d$; +4. the donor is entered into the commit queue as if newly registering, except that its recorded block height is the one set by the `deregister` call, and its target is fixed to the sibling prefix of the departing node; +5. the donor mines a new conforming overlay and activates it within the validity window; and +6. upon the donor's activation, the departing assignment is removed and the departure completes. + +The donor MAY operate its old and new overlays concurrently during the transition. + +If the donor fails to activate within the validity window, it is dropped from the registry and its stake is forfeited; a fresh donor is drawn with new entropy and the procedure repeats. The departing node remains active until a donor completes. + +#### Data handover + +This SWIP specifies only the assignment of nodes to neighbourhoods; the handover of stored content during relocation is deliberately left unspecified. A relocated node is expected to synchronise the reserve of its newly assigned neighbourhood, while the neighbourhood it vacated retains the data with the sibling node. With upcoming durability guarantees, such relocation gaps become repairable events; in particular, a newly assigned node need not sync its reserve live from its sibling but may instead acquire the neighbourhood's content from cold storage. + +### Randomness and anti-grinding measures + +For request $q$, registered in block $h_q$, let the entropy block be: $$ -o_n \notin \mathit{nh}_{k, S'} +t_q=h_q+\Delta, $$ -which entails that node $k$'s neighbourhood does change between $S$ and $S'$ and: +where $\Delta\geq1$ is a configured delay. The seed becomes computable only after $t_q$ has the configured number of confirmations, and remains computable only while its block hash is available to the contract. + +The seed is domain-separated: $$ -\mathit{nh}_{k, S'}\subset \mathit{nh}_{k, S}. +\rho_q=\operatorname{keccak256}( +\texttt{"SWIP39"}\parallel +\operatorname{chainID}\parallel +\operatorname{contractAddress}\parallel +q\parallel +a_q\parallel +\operatorname{blockhash}(t_q)). $$ -This can only happen if $k$'s uniquness depth increases, from which it follows that, for set $S$, +The deployment MUST specify $\Delta$, the confirmation delay, and the validity window. On an EVM deployment using the `BLOCKHASH` opcode, activation MUST occur before the referenced hash becomes unavailable. + +The block hash is unpredictable to the applicant at registration under the assumed chain model, but it is not a perfect randomness beacon. A proposer may have limited influence through block construction, withholding, or censorship. Deployments requiring stronger guarantees SHOULD replace the block-hash source with a verifiable randomness mechanism while retaining the same domain separation and request binding. + +Selection MUST be uniform over the eligible count $m$. Directly computing $\rho\bmod m$ introduces modulo bias unless $m$ divides $2^{256}$. Implementations MUST use rejection sampling: $$ -u_k = d, +L=2^{256}-(2^{256}\bmod m). $$ -and for $S'$, +Interpret successive domain-separated hashes as integers until $x < L$, then select rank: $$ -u_k = d+1. +r=x\bmod m. $$ -In other words, any node added to $S$ must join a free neighbourhood $\mathit{nh}_{k,S}$ but _must not match the_ $d+1$-_th bit of_ $o_k$. +The following measures limit assignment grinding: -Conversely, when a node $g$ is removed, it must have a sister so that after removal, that balancedness remains invariant. This may necessitate rebalancing first, i.e., in case $g$'s removal would result in a non-balanced set, requires finding a random donor $j$ with $u_j=d+1$ to be reinserted as $g$'s sister.[^donor] +- stake is locked before entropy is known; +- an identity has at most one live registration; +- the application fee is not refunded after entropy becomes available; +- an expired registration forfeits its fee; +- request IDs and identities are included in the seed; and +- selection is computed from contract state, not from a caller-supplied list. -[^donor]: In fact, the donor could just reinsert at $g$'s spot, however, any short overlap of activity is easier to handle if donor is $g$'s sister. +These measures price selective aborts but do not prevent one operator from funding multiple identities. Security analysis MUST therefore express resistance in economic and probabilistic terms rather than as a one-operator-one-node guarantee. -## Architecture +### Contract state and transitions -This contract is deployed together with a staking contract similar to the [swarm storage incentive staking contract](https://github.com/ethersphere/storage-incentives/blob/master/src/Staking.sol). This contract will retain the total stake treasury, as well as enabling a node operator to deposit, withdraw and maintain their stake. Concerns should be strictly separated to improve security of locked funds and upgradability of both contracts. +The registry is deployed alongside the current staking contract (cf. the [swarm storage incentive staking contract](https://github.com/ethersphere/storage-incentives/blob/master/src/Staking.sol)), and it is the staking contract that uses the registry: assignment, balance bookkeeping, and donor selection live here, while custody of stake and the freezing and slashing logic driven by the redistribution game remain the staking contract's concern. The registry only reports assignment events — activation, expiry, donor default — and never holds or slashes stake itself. This strict separation of concerns improves both the security of locked funds and the upgradability of either contract. -The node assignment contract is composed of several transactional endpoints: +The registry maintains: -$$ -\begin{array}{lllll} -\hline %toprule -& \text{HTTP request} & & \text{HTTP response}\\ -% \multicolumn{3}[c]{HTTP request} & \multicolumn{2}[c]{HTTP response}\\ -\text{method} & \text{URL} & \text{description} & \text{status} & \text{body} \\ -\hline %midrule -\texttt{POST} & \texttt{/bzz/bal/} & \text{registration} & 201, 400, 402, 404 & \\ -\texttt{PUT} & \texttt{/bzz/bal/assign/}\langle\texttt{overlay}\rangle & \text{assign overlay} & 200, 400, 404 & \text{overlay prefix constraint}\\ -\texttt{POST} & \texttt{/bzz/bal} & \text{deregister} & 200, 201, 204, 400, 402, 404 & \text{pending rebalancing}\\ -\texttt{DELETE} & \texttt{/bzz/bal/unassign} & \text{leave} & 200, 400, 404 \\ -\hline %bottomrule -\end{array} -$$ +- `activeCount`: the number $N$ of active nodes; +- `leaves`: trie leaves containing owner, public-key commitment, and overlay; +- `participants`: staking identity to active assignment; +- `registrations`: request ID to registration record; +- `registrationQueue` ($C_R$) and `departureQueue` ($C_D$): pending request IDs in commit order; +- subtree counts for split-candidate and donor-pair selection; +- `pendingDeparture`: the departure currently awaiting donor relocation, if any; and +- configuration parameters and references to the staking contract. -### Registering and Random Assignment +Joins require no coordination state: registrations queue freely and activations validate against current state. Only donor relocation pends: -Candidate nodes end up assigned to a random free neighbourhood in a way that all the potential free neighbourhoods had the same chance of being selected. +| State | Meaning | Permitted terminal action | +|---|---|---| +| `Idle` | No pending rebalancing | Registrations, activations, and direct departures proceed freely | +| `DonorPending` | Donor drawn, relocation in progress | Donor activates and the departure completes, or the deadline passes: stake forfeited, fresh donor drawn | -#### Registration +```mermaid +stateDiagram-v2 + [*] --> Idle + Idle --> Idle: register / activate / direct deregister + Idle --> DonorPending: deregister requiring rebalancing + DonorPending --> Idle: donor activates, departure completes + DonorPending --> DonorPending: deadline passed, stake forfeited, fresh donor drawn +``` -In the first step, a node's intention to participate as a provider in the service network gets recorded in the _commit queue_ $C$. -The current blockheight $h_i$ is recorded together with the ether address $a_i$ by pushing an _entry struct_ ($e_i = \langle a_i,h_i\rangle$) to the end of commit queue. +Contract methods and Bee HTTP endpoints are distinct interfaces. A reference contract is expected to expose operations equivalent to: -At the time of registering, we check if the node's ether address is not already on the list. -In order to prevent repeated trials, each node must be registered only once. -A non-refundable application fee is deposited. +- `register(publicKeyCommitment)`; +- `targetPrefix(identity)` (read-only); +- `activate(requestID, nonce, authorization)`; +- `expire()`; +- `deregister()`; +- `expireDonor(leaveID)`; +- `getAssignment(identity)`; and +- `closestActiveNode(address)`. -#### Expiry +All state-changing methods MUST validate the expected state and deadline. External calls to staking or proof-verification contracts MUST follow checks-effects-interactions or an equivalent reentrancy-safe pattern. -The entry is valid for a period of $G$ blocks after the registration. In practice, $G$ must be less than $256$, the number of blocks for which the blockhash is available from within the EVM. +## Implementation -Since the blockheight values of the commit queue items are monotonically increasing, entries at the beginning of the list expire first. By iterating up to the first valid entry, expired entries can be iterated on efficiently. +This section describes one implementation strategy. The invariant and state transitions are normative; the exact storage layout is not. -The `expire` function call iterates the commit queue from the oldest, going through all expired entries, burns their deposit, and, by setting the head of the list to the first valid item, removes them from the front of the commit queue. +### Trie representation and algorithms -After calling `expire`, the validity of the registration is checked by finding the entry for the ether address in the commit queue. +An _implicit complete binary trie (ICBT)_ uses one-based heap indices; no pointers are stored, and the represented tree is traversed using basic arithmetic on the indexes. The root has index $1$; each trie node corresponds to a neighbourhood, with the shared prefix expressed by the traversal (left is 0, right is 1): -#### Entropy +| Description | Notation | Definition | +|---|---|---| +| depth of $i$ | $\operatorname{Depth}(i)$ | $\lfloor\log_2 i\rfloor$ | +| parent of $i$ | $\operatorname{Parent}(i)$ | $\lfloor i/2\rfloor$ | +| left child of $i$ | $\operatorname{Left}(i)$ | $2i$ | +| right child of $i$ | $\operatorname{Right}(i)$ | $2i+1$ | +| sibling of $i$ | $\operatorname{Sibling}(i)$ | $i+1-2(i\bmod 2)$ | +| position of $i$ within its level | $\operatorname{Position}(i)$ | $i-2^{\operatorname{Depth}(i)}$ | +| index of prefix $p$ with $\lvert p\rvert=\ell$ | $\operatorname{Index}(p)$ | $2^\ell+\operatorname{uint}(p)$ | -Nodes derive randomness from a high entropy seed +The binary representation of $\operatorname{Position}(i)$, padded to $\operatorname{Depth}(i)$ bits, is exactly the prefix designated by index $i$ — so the constraint prefix handed to the miner can be read off the index alone, with no storage access. The first three levels: -$$ -\rho_i = \mathit{H}(\text{blockhash}(h_i+1) \parallel h_i \parallel a_i), -$$ +```mermaid +graph TD + 1["1: ε"] + 1 --> 2["2: 0"] + 1 --> 3["3: 1"] + 2 --> 4["4: 00"] + 2 --> 5["5: 01"] + 3 --> 6["6: 10"] + 3 --> 7["7: 11"] + 4 --> 8["8: 000"] + 4 --> 9["9: 001"] + 5 --> 10["10: 010"] + 5 --> 11["11: 011"] + 6 --> 12["12: 100"] + 6 --> 13["13: 101"] + 7 --> 14["14: 110"] + 7 --> 15["15: 111"] +``` -which is not known at the time of registration. The _validity window_ $VW < 256$ ensures that the referenced blockhash remains accessible. +Each stored trie record may contain: -#### Neighbourhood assignment +- an active leaf assignment, if one exists at that exact prefix; +- `splitCount`, the number of eligible split candidates in its subtree; and +- `donorCount`, the number of eligible donor-pair parents in its subtree. -Assignment of a node to a neighbourhood must observe balancedness of the set as an invariant. Since -$2^{d+1} - N$ is the number of free neighbourhoods. A node computes +Only nodes on a modified leaf-to-root path require counter updates, giving $O(\log N)$ tree writes per activation or completed departure. The stored trie is sparse and requires $O(N)$ assignment records. -$$ -r_i = \rho_i \bmod (2^{d+1}-N). -$$ +The ICBT SHOULD be implemented as a self-contained container data-structure contract (or library) exposing only the generic trie operations — insert, remove, ranked selection, counter maintenance, and closest-match lookup — which the balancing registry contract then merely uses. This separation keeps the data structure independently testable against a reference model, reusable by other registries, and auditable in isolation from the protocol logic. -Let $f_0, f_1, \ldots, f_{F-1}$ be the indexes of free neighbourhoods, these are available for assignment. Out of these free neighbourhoods, one is chosen with equal probability: +#### Rank selection -$$ -k=f_{r_i} -$$ +Given a zero-based rank $r < c(1)$, where $c(i)$ is the relevant aggregate count for the subtree rooted at $i$, selection descends as follows: + +```text +selectByRank(r, count): + i = 1 + while i is not an eligible terminal: + left = 2 * i + if r < count(left): + i = left + else: + r = r - count(left) + i = left + 1 + return i +``` -This calculation is made available as a read-only call. It takes as argument a node's ether address $a_i$ returns the current prefix constraining the overlay assignment.[^format] +Selection is read-only. Counters are updated only when an assignment activates, a departure completes, or a donor is dropped. The implementation MUST NOT decrement aggregate counters merely because a view function was called. + +The target prefix can be accumulated bit by bit during the same descent — shift left on every step, set the low bit when going right — and extended by one final bit chosen as the complement of the incumbent's next overlay bit, so that the applicant takes the free child: + +```text +targetPrefix(r): + i = 1 + c = 0 # accumulated prefix bits + while Depth(i) < d: + c = c << 1 + i = Left(i) + if r >= splitCount(i): # not enough candidates on the left + r = r - splitCount(i) + i = i + 1 # continue in the right subtree + c = c | 1 + b = overlay(i)[d] # incumbent's next bit + return (c << 1) | (1 - b) # applicant takes the sibling child +``` + +Since the prefix equals the binary representation of $\operatorname{Position}(i)$, the accumulator `c` is redundant with the final index — it is shown for clarity; an implementation may just return $\operatorname{Position}$ of the selected child. -[^format]: the neighbourhood constraint prefix is formatted the same way as when given to the bee client as the target of mining: `010101010101` +Donor selection descends the same way over `donorCount`, terminating at a depth-$d$ parent whose two children are active leaves, and then consumes one further random bit to pick the moving donor from the pair: -Calling the function multiple times may return a different constraint prefix if there was another successful assignment in between the two calls. +```text +selectDonor(r, s): # rank r over donor pairs, s one extra seed bit + i = 1 + while Depth(i) < d: + i = Left(i) + if r >= donorCount(i): # not enough pairs on the left + r = r - donorCount(i) + i = i + 1 # continue in the right subtree + j = Left(i) + s # choose one of the two siblings uniformly + return leaf(j) # the moving donor +``` -#### Mining an overlay nonce +The bit $s$ is taken from the same seed $\rho_q$ (e.g. the next bit after those consumed by rejection sampling), so the choice within the pair is as unpredictable as the pair itself. The donor's sibling $\operatorname{Sibling}(j)$ takes over the pair's parent prefix $i$, which is what makes the donor's removal a balanced one. -The commited node, upon learning the neighbourhood $k$, must find (mine) a nonce $\nu_i$ to generate its actual _overlay address_ +#### Commit queues and expiry + +Both queues $C_R$ and $C_D$ hold entry structs $e=\langle a,h\rangle$ — the committing identity and its commit block height — as append-only arrays with a head index. Since block heights are recorded in commit order, they are monotonically increasing, and expired entries are always contiguous at the front. `expire` therefore never scans the whole queue: it advances the head past entries older than the validity window, burning their fees, and stops at the first valid entry: + +```text +expire(queue, maxSteps): + steps = 0 + while queue.head < queue.length and steps < maxSteps: + e = queue[queue.head] + if e.height + VW >= currentBlock: break # first valid entry + burnFee(e) + queue.head = queue.head + 1 + steps = steps + 1 +``` + +This is $O(1)$ amortized and bounded per call by `maxSteps`. It SHOULD be invoked at the start of `activate` and `deregister` processing, so the queues are clean before any state transition; after `expire`, checking that a registration is live reduces to verifying that its entry sits at or beyond the head. + +A donor's re-entry is pushed with the block height recorded by the `deregister` call, so it expires on the same schedule as the departure that caused it. A defaulted donor's entry is passed over like any expired entry — its stake forfeited by `expireDonor`, which pushes the fresh donor's entry at the current height. + +For $N\geq1$, joining terminals are active leaves at depth $d$. Donor-selection terminals are depth-$d$ parents whose two children are active leaves. The contract MUST assert before selection that the root count equals the count implied by the invariant: $$ -o_i := \mathit{H}( \nu_i \parallel a_i \parallel networkID ) +\operatorname{splitCount}(1)=2^{d+1}-N, $$ -that falls in the correct neighbourhood. - $$ -\nu_i \leftarrow \mathbb{\Sigma}^{256},\text{ such that } o_i\gg(255-d)=k \land o_i\ll d\gg 255 = O![k]\ll d\gg 255 +\operatorname{donorCount}(1)=N-2^d. $$ -#### Assign the actual overlay +For $N=0$, the implementation uses the empty prefix as a single bootstrap slot and has no donor pair. -The assign transaction call is the second transactional endpoint called by the staking contract. It takes the provider's ether address as well as the mined overlay as arguments. -The mined overlay $o_i$ must be submitted to the contract, which, once the overlay is verified, removes the entry from the commit queue and then records the complete overlay $o_i$ associated to the node's ether address $a_i$ (it does make a difference what data structure is used). +These assertions provide useful invariant checks in tests even if production code omits them for gas reasons. -### deregistration and Rebalancing +### Client integration -Nodes are free to deregister at any time. -If the sister node exists, removal proceeds directly and the invariant remains satisfied. +Bee requires an operator-facing workflow for both initial assignment and relocation: -If removal would leave both children of the parent empty, then _rebalancing_ is required: upon calling `deregister`, one _donor pair_ of nodes is selected from among the full (doubly filled) neighbourhoods of depth $d$. From the selected pair, one of the two nodes is chosen as _donor_. The donor's neighbourhood is taken over by its sister, so the donor is removed first exactly as if it had chosen to leave itself — balance is preserved, since the sister becomes unique at depth $d$. The donor is then entered into the commit queue as if newly registering, except that the recorded blockheight is set by the `deregister` call, and it gets assigned the sister neighbourhood of the node to be removed. The donor must complete this relocation within the validity window; otherwise it is dropped from the registry with its stake forfeited, and a fresh donor is drawn. +1. create or load the node key; +2. register its public-key commitment through the staking/registry integration; +3. wait for the entropy delay; +4. query the current target prefix and deadline; +5. mine an overlay nonce satisfying that prefix; +6. submit activation (re-querying and re-mining if the target changed); +7. start or update the node with the activated overlay; and +8. expose registration and relocation progress. -The original node is removed only after the donor successfully completes reassignment, ensuring that the invariant is never violated. In order that the rebalancing cannot be manipulated, i.e., the selected node reinserted into the neighbourhood of the deregistrant, the donor must be selected with proper randomness, not known at the time of deregistration. This randomness is derived the same way as when we add a node: deregistration call just records the block height $h_i$, and the following blockhash serves as the high entropy seed for randomness: +The contract and Bee MUST use exactly the same overlay derivation function, field encoding, network ID, and bit ordering. These values MUST be fixed by test vectors before deployment. -$$ -\rho_i = H(\text{blockhash}(h_i+1) \parallel h_i \parallel a_i), -$$ +For relocation, Bee SHOULD support a staging mode in which the old overlay continues serving while the new overlay is mined and synchronized. What data moves, and how, is outside this SWIP (see [Data handover](#data-handover)). -Given the number of free neighbourhoods currently full (doubly filled) is $2N-2^{d+1}$. A node computes +Read-only Bee endpoints MAY expose registry state, but they MUST NOT be confused with contract methods. Suggested client operations are: -$$ -r_i = \rho_i \bmod (N-2^{d}) -$$ +- begin or inspect registration; +- report the current target prefix and expiry; +- begin nonce mining; +- report mining progress; +- report synchronization status; and +- finalize activation or departure. -Let $f_0, f_1, \ldots f_{N-2^{d}-1}$ be the sorted list of indexes for doubly-filled (non-free) neighbourhoods, i.e., those whose continuations are both unique at $d+1$. +## Security -#### Data handover +The registry improves distribution only under explicit identity, randomness, and chain-finality assumptions. It must be evaluated together with staking and the storage-proof or service protocol. -This SWIP specifies only the assignment of nodes to neighbourhoods; the handover of stored content during relocation is deliberately left unspecified. A relocated node is expected to synchronise the reserve of its newly assigned neighbourhood, while the neighbourhood it vacated retains the data with the sister node. With upcoming durability guarantees, such relocation gaps become repairable events; in particular, a newly assigned node need not sync its reserve live from its sister but may instead acquire the neighbourhood's content from cold storage. +### Threat model -## Specification +The adversary may: -### Data Structures +- create and fund multiple Ethereum identities; +- operate multiple Bee nodes; +- abandon an assignment after learning its prefix; +- reorder, front-run, or censor transactions when it controls block production; +- attempt to bias a future block hash; +- cause nodes to go offline during joins or relocations; +- submit stale state or invalid overlay nonces; +- target contract operations with large queues or expensive cleanup; +- exploit reentrancy or inconsistent cross-contract state; and +- withhold stored data after relocating. -For registration and deregistration the contract maintains two distinct commit queues $C_R$ and $C_D$. -This initially empty list of _entry_ struct types holds information about the node that committed to enter or leave the DSN. The entry struct holds information about the ether address of the node and the blockheight the address registered at. +The adversary is not assumed to: -The queues are always extended by the `(de)register` functions (tx-s) and cleaned by `expire` (called by `(un)assign`). +- break the hash function or signature scheme; +- predict an honest future entropy source before registration; or +- revert finalized chain history beyond the deployment's confirmation assumption. -$$ -C_X: \texttt{[}\,\texttt{]}\ \mathit{entry} -$$ +#### Sybil concentration -Mappings from Ethereum address ($A$) to are maintained: +If there are $m$ equally eligible split candidates and an attacker finalizes $q$ independent registrations, the number assigned to any particular candidate follows a binomial distribution with success probability $1/m$, subject to the tree changing after each successful activation. The exact multi-round probability depends on those state changes. -$$ -A : \mathbb{\Sigma}^{160} \mapsto \mathbb{N}. -$$ +The proposal does not impose a global-majority threshold below which concentration is impossible. Instead, each additional trial requires capital and exposes the attacker to a non-refundable fee or to operating the node at its assigned prefix. Deployment analysis SHOULD estimate the cost of obtaining a target concentration at expected network size. -Given an active set of nodes $S$, such that for any node with address $a$, $A(a)=N=\mid S\mid$ where $N$ is the number of nodes in $S$ when (ie just before) $n$ is inserted. In other words, if $A(a)=n$ then node with address $a$ was inserted as the $n$-th node. +#### Assignment races -### ICBT +Because no reservation is held, two applicants may compute the same target, or an activation may change the tree while another applicant mines. The activation transaction validates against current state, so consistency is never at risk; the cost is wasted mining work for the loser, who recomputes and re-mines. Under realistic churn rates such collisions are rare, and their cost falls entirely on the colliding applicants. -The association of neighbourhoods and nodes is stored in a data structure we call _implicit complete binary trie (ICBT)_. Each node $v$ of the trie corresponds to a neighbourhood (the shared prefix is expressed by the traversal, i.e., left is 0, right is 1). +#### Selective abort and liveness -#### Layout +A participant can refuse to activate an undesirable target, and a selected donor can refuse to relocate. Fees, deadlines, and stake forfeiture make these choices costly, while permissionless expiry and redraw calls restore progress. Since joins are not serialized, a stalling applicant delays nobody but itself; a stalling donor delays only the departure it was drawn for, bounded by the validity window, after which its stake is forfeited and a fresh donor is drawn. -This data structure has the role of maintaining the number of neighbourhoods that are (free = assignable to a new peer, or full = selectable as donor) under a node. +#### Entropy manipulation -$$ -I : \mathbb{N}\to \mathit{node} \cup \{\varnothing\}, -$$ +Future block hashes provide applicant unpredictability, not cryptographic proof of unbiased randomness. Confirmation delays reduce reorganization risk but do not remove proposer influence. High-value deployments SHOULD use a stronger beacon or verifiable randomness function. -where $I[i] = \varnothing$ (nil value) denotes an empty slot and the node value type is a tuple of the +#### Contract denial of service -- overlay, -- the number of free/full slots subsumed under the subtree, -- and the ordinal of first insertion +Queue processing MUST be bounded per transaction. Expiry SHOULD advance a queue head or process a caller-specified maximum number of entries; it MUST NOT require iterating through all historical registrations. -$$ -\mathit{node}= \langle \Sigma^{256}, \mathbb{N}, \mathbb{N}\rangle -$$ +No transition should require iteration over all active leaves. Ranked subtree selection and path updates keep work logarithmic in $N$. -Both the second and third is constrained to $\overline{0,N-1}$. +#### Storage handover -Since the values of $A$ form a continuous range of integers from 0 to N (i.e., unique ordinal of first insertion), it is possible to use this as a type-agnostic version of this data structure and maintain a typed array mapping from the third component of the node struct, to another one. +The registry tracks the assignment of address ranges, not possession of data. Whether and how stored content moves when assignments change is outside this SWIP (see [Data handover](#data-handover)): the vacated neighbourhood retains the data with the sibling node, and durability guarantees make relocation gaps repairable events. -If $A(a)=n$ then its index in the trie is the depth at which it is unique ($i=u_i-2^d$) +#### Shadow-world fabrication -$$ -I[i]=\langle o_i, 0, N \rangle. -$$ +Balanced assignment can force an attacker seeking control of selected neighbourhoods to acquire assignments across the global prefix space. It does not by itself prove that the attacker's stored view matches the live network or that advertised stamp utilization is genuine. -Now an illustration: +Any claimed cost ratio for a shadow-world attack depends on the postage-game sampling rule, utilization measurement, reward size, and the attacker's ability to reuse data. Those assumptions and the resulting derivation SHOULD be specified in the postage redistribution SWIP rather than asserted here without a model. -```mermaid -graph TD - R((root)) - 0 --> 1 - 1 --> 2 - 1 --> 3 - 2 --> 4 - 2 --> 5 - 3 --> 6 - 3 --> 7 - 4 --> 8 - 4 --> 9 - 5 --> 10 - 5 --> 11 - 6 --> 12 - 6 --> 13 - 7 --> 14 - 7 --> 15 -``` +### Gas and performance analysis -The implicit binary structure means the represented tree can be traversed using basic arithmetic on the indexes: +Let $d=\lfloor\log_2 N\rfloor$. -$$ -\begin{array}{l|l|l} -\mathrm{description} & \mathrm{notation} & \mathrm{definition}\\\hline -\text{parent of }i& \mathrm{Parent}(i) & i/2 \\ -\text{left child of }i&\mathrm{Left}(i) & 2i\\ -\text{right child of }i& \mathrm{Right}(i) & 2i+1\\ -\text{sister of }i& \mathrm{Sister}(i) & i + 1 - 2 (i \bmod 2)\\ -\text{depth of }i& \mathrm{Depth}(i) & \mathrm{Floor}(\log_2(i))\\ -\text{position of }i& \mathrm{Pos}(i) & i \mod 2^{\mathrm{Depth}(i)} -\end{array} -$$ +- Selecting a split candidate or donor pair takes $O(d)$ trie reads. +- Activating a join or completing a departure updates $O(d)$ aggregate records. +- Active assignment storage is $O(N)$. +- Registration and expiry are $O(1)$ amortized when implemented with mappings and a monotonic queue head. +- Overlay mining for an $\ell$-bit prefix requires $2^\ell$ hashes in expectation. -#### Resolve values via parents +Since $\ell$ is approximately $\log_2N+1$, expected overlay-mining work grows approximately linearly with the number of registered nodes. This off-chain cost is a material scalability constraint and MUST be benchmarked for target network sizes and supported hardware. -When the index structure is used as a map, the rule of inheritance allows you to look up a value that was 'inherited' from an earlier stage (inserted at a shallower depth). We can define $V$ as a lookup function for a map over the above index structure, then $V!$ is +Donor relocation adds nonce mining bounded by the validity window; its completion time is governed by mining and transaction finalization, not by data transfer, which is out of scope. -$$ -V!(i)=\begin{cases} -V!(\mathrm{Parent}(i)) &\text{if } V(i)=\varnothing\text{ and }i>1\\ -V(i) &\text{otherwise} -\end{cases} -$$ +The reference implementation SHOULD report gas for: -We can define the predicate _not assigned_ ($\mathit{NA}$) as follows: +- registration; +- activation at several tree depths; +- direct departure; +- donor selection; +- donor activation; +- expiry and redraw. -$$ -\mathrm{NA}(i) \Longleftrightarrow V(i) = \varnothing . -$$ +## Migration and backwards compatibility -This allows us to define free and fully assigned neighbourhoods: +Deployment requires coordinated changes to the staking contract, registry contract, and swarm node client software. Existing nodes have self-selected overlays and cannot simply be inserted into a balanced registry without either relocation or an explicitly temporary imbalance. -$$ -\mathrm{Free}(i) \Longleftrightarrow \mathrm{NA}(\mathrm{Left}(i)) \lor \mathrm{NA}(\mathrm{Right}(i)) -$$ +A migration plan MUST specify: -and +1. the snapshot or eligibility rule for existing stake; +2. whether existing nodes retain their identities and keys; +3. how initial prefixes are assigned; +4. the time allowed to mine new overlays and synchronize reserves; +5. whether old and new redistribution games overlap; +6. how stake moves between old and new contracts; +7. rollback conditions; and +8. the block or condition at which the new registry becomes authoritative. -$$ -\mathrm{Full}(i) \Longleftrightarrow !\mathrm{NA}(\mathrm{Left}(i)) \land !\mathrm{NA}(\mathrm{Right}(i)) -$$ +A safe staged migration is: -The data structure operations all enforce the condition +1. deploy and audit the new staking and registry contracts; +2. register existing operators without yet making the registry authoritative; +3. assign prefixes and allow nodes to mine overlays; +4. synchronize new reserves while old overlays remain active; +5. verify minimum coverage and client readiness; +6. activate the new game at a declared boundary; and +7. retire old assignments only after the transition criteria are met. -$$ -\forall i, \mathrm{Depth}(i)< d\Rightarrow V(i)\neq \varnothing \lor V(i+1)\neq \varnothing -$$ +Fixed stake per node is recommended if reward eligibility is intended to correspond directly to the number of independently operated reserves. If variable stake remains supported, its interaction with random assignment and expected revenue MUST be specified separately. -which ensures that every prefix of length $d-1$ contains at least one node. This invariant constrains the admissible states of the system. +## Worked examples -The ICBT is used to +Let $\epsilon$ denote the empty prefix. One possible balanced progression is: -- find candidate neighbourhoods for new applicants -- find candidate donors for rebalancing -- find the closest node to an address (prefix) +| $N$ | $d$ | Active leaf prefixes | +|---:|---:|---| +| 1 | 0 | $\epsilon$ | +| 2 | 1 | $0,1$ | +| 3 | 1 | $00,01,1$ | +| 4 | 2 | $00,01,10,11$ | +| 5 | 2 | $000,001,01,10,11$ | +| 6 | 2 | $000,001,010,011,10,11$ | +| 7 | 2 | $000,001,010,011,100,101,11$ | +| 8 | 3 | $000,001,010,011,100,101,110,111$ | -### Counting free neighbourhoods for candidate assignment +Every row is prefix-free and covers the full address space. For $N=5$, three leaves have depth $2$, two leaves have depth $3$, and the largest area is twice the smallest. -Function $F_0(i)$ on the index tracks the number of free slots (candidate neighbourhoods to assign) in the left subtree rooted at index $i$: +The five-node state can be visualized as: -$$ -F_0: \mathbb{N}\to\mathbb{N}\\ -F_0(i) = \begin{cases} -F_0(Left(i)) + F_0(Right(i))&\text{if } Depth(i) p0["0"] + root --> p1["1"] + p0 --> p00["00"] + p0 --> p01["01: active"] + p00 --> p000["000: active"] + p00 --> p001["001: active"] + p1 --> p10["10: active"] + p1 --> p11["11: active"] +``` + +Only the labelled active leaves form the partition. Internal prefixes such as $0$, $00$, and $1$ are not assignments. + +### Join from five to six nodes -When the trie becomes fully balanced with a number of nodes turning $N = 2^D$, then each neighbourhood at level $D$ is free, i.e., has exactly one assignable child: +Assume the active leaves are: $$ -\forall 2^{D-1}\leq i< 2^{D}, \quad F_0(i)=1 +\{000,001,01,10,11\}. $$ -In this case, +Here $N=5$, $d=2$, and the split candidates are: $$ -\forall 0< i<2^{D-1}, \quad F_0(i) = 2^{D-Depth(i)}. +\{01,10,11\}. $$ -By the time the next depth is reached, $N=2^{D+1}-1$-th element is assigned, all -of the free neighbourhoods got allocated, thus: +Thus: $$ -\forall 2^{D}\leq i< 2^{D+1}, \quad F_0(i)=0 +\operatorname{splitCount}(1)=2^{3}-5=3. $$ -and therefore, +Suppose uniform selection yields leaf $10$, and the incumbent overlay starts with $100$. The incumbent keeps child $100$, while the applicant targets $101$. After activation: $$ -\forall 0< i<2^{D}, \quad F_0(i) = 0. +\{000,001,01,100,101,11\}. $$ -We define now a sketch of the _OverlayPref_ function. its single argument is $r_i$. +There are now two depth-$2$ leaves and four depth-$3$ leaves, matching: -```python -func OverlayPref( k uint32 ) returns []byte - uint32 j = 1 - uint32 c = 0 - for depth(j)<=depth() - if !NA(O(j)) o=O(j) # overlay assigned - c<<=1 # shift c - j=Left(j) # set j to left child - if k>F(j) # if k over volume on left branch - k-=F(j) # then adjust to subtree volume; - j++ # set j as the right child - c||=0x01 # right - F(j)=F(j)-1 # decrement volume - c<<=1 - if o[d+1] == 0 # must not match existing - c||=0x01 # - return rbinary.bigendian(c)[ 31-d : ] -``` +$$ +2^{3}-6=2, +\qquad +2(6)-2^3=4. +$$ -### Counting fully saturated leaves for donor selection +### Direct departure from five to four nodes -The second quantity one needs to track is the number of nodes in the subtree with both their children assigned on level ${d}$: these correspond to _candidate donor pairs_. -We can use $F_1$ +Start from: $$ -F_1: \mathbb{N}\to\mathbb{N}\\ -F_1(i) = \begin{cases} -F_1(Left(i)) + F_1(Right(i))&\text{if } Depth(i)|"r=1 ≥ 1: skip, r←0"| 2 + 1 ==>|"go right"| 3 + 3 ==>|"r=0 < 1: go left"| 6 + 3 -.-> 7 +``` - uint32 j = 1 - for depth(j)F(j) # if k over volume on left branch - k-=F(j) # then adjust to subtree volume; - j++ # set j as the right child - F(j)=F(j)-1 # decrement volume +Solid arrows mark the traversal; the dashed branches are only inspected for their counts. The incumbent at $10$ has next bit $0$ (overlay starts $100$), so `targetPrefix` returns $10\mathbin\Vert 1=101$. - if !NA(O(j)) o=O(j) # depth overlay assigned - j=Left(j) # set j to left child - j+=rand(2) # set to right with 0.5 probability - if !NA(O(j)) o=O(j) # depth overlay assigned +## Reference implementation - return o -``` +A conforming reference implementation should include: -### Further endpoints +- a registry contract implementing the state machine; +- a staking-contract adapter; +- a Swarm client integration for registration, nonce mining, and relocation; +- deterministic overlay-derivation test vectors; +- property tests for the balance invariant; +- adversarial tests for expiry, redraw, selective aborts, and queue growth; +- gas benchmarks across representative values of $N$; and +- an executable migration test. -A public read-only endpoint exists for querying neighbourhoods as well as nodes. Accessor for $d$ and $N$ will return the current neighbourhood depth and the current number of assigned neighbourhoods. A public accessor for $A_d$ will return for a neighbourhood (between $0$ and $2^d-1$ inclusive) the overlay of the node assigned to that neighbourhood. Another endpoint will return for any partial address $o$ the closest node, so that the network service can find responsible nodes for (i.e., closest to) any address in the space shared by overlays: +At minimum, property-based tests MUST verify after every completed transition that: $$ -\begin{eqnarray} -g&:&\Sigma^{256}\to\Sigma^{256}\\ -g(a)&=&O!(a\gg(254-d)) -\end{eqnarray} +|L(S)|=N, $$ -If the resulting overlay address falls into the neighbourhood that the registrant was assigned to, i.e., the correctness of the nonce submitted from the perspective of the staking contract. +the sub-network has full coverage: -### Changes to the bee client +$$ +\bigcup_{p\in L(S)}NH(p)=\Sigma^{256}, +$$ + +nodes have disjoint neighbourhoods: -A new endpoint must be added to clients to register a node that is not yet registered to be assigned a neighbourhood. Once the neighbourhood is known, the client can mine the nonce needed to place the overlay in the required neighbourhood. +$$ +\forall p,q\in L(S),\ p\neq q +\Longrightarrow NH(p)\cap NH(q)=\varnothing, +$$ + +the nodes are balanced: + +$$ +\forall p\in L(S),\ |p|\in\{d,d+1\}, +$$ -### Migration +and that every active overlay starts with its assigned prefix. -Since a new updated staking contract, a stake migration will be needed for the upgrade. Before the change, all the simplification of the staking contract is recommended, especially to allow fixed stake in order to realign redundancy of storage and monetary incentive: with a fixed amount staked, total stake is linearly proportional to the number of nodes, and therefore comparisons across neighbourhoods can be made based on the number of nodes. In particular, the random balanced assignment makes sense in terms of incentives (expected revenue). +The implementation SHOULD also generate random valid sequences of joins, direct departures, donor relocations, expiries, and redraws, comparing contract state against a simple off-chain model after each completed transition. +The exact Solidity ABI, Bee API paths, economic parameters, and deployment addresses remain to be supplied before this SWIP can advance beyond Draft. From 98d1a950ce11359970038c0a18e9fa9e0a3a786c Mon Sep 17 00:00:00 2001 From: zelig Date: Mon, 27 Jul 2026 03:24:27 +0200 Subject: [PATCH 22/23] address review feedback: merge scope+goals, rework states, simplify API - Merge Scope and Goals into one 'Scope and goals' chapter with a single terminology section - State machine: rename Idle to Active, add explicit Expired state for a donor that missed its relocation deadline - Drop request IDs: registrations keyed by staking identity (one live registration per identity); seed derived from identity and entropy block alone - Allow any number of concurrent pending departures, each with its own donor and deadline; pending-departure leaves excluded from split candidates - Shorten public API: target, activate, expireReg, deregister, expireDereg, getPrefix, nodeFor - Replace all Bee references with Swarm / swarm node client terminology Co-Authored-By: Claude Fable 5 --- SWIPs/swip-39.md | 142 +++++++++++++++++++++++------------------------ 1 file changed, 68 insertions(+), 74 deletions(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index cd2c8e5b..97ed394d 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -17,24 +17,38 @@ A node joins in exactly two transactions: it first commits its identity, stake, The mechanism is intended to make targeted concentration in a particular neighbourhood probabilistic and costly, and to support balanced allocation of storage-incentive or decentralised-service workloads. It does not establish operator identity or prevent one operator from registering multiple nodes. -## Scope +## Scope and goals -This SWIP defines: - -- the balance invariant maintained by the registry; -- the join, activation, and departure state transitions; -- the source and use of randomness; -- the contract state required to select prefixes and donor nodes; -- the integration points required by the swarm client; and -- the security and liveness assumptions of the mechanism. +This SWIP defines the balance invariant maintained by the registry; the join, activation, and departure state transitions; the source and use of randomness; the contract state required to select prefixes and donor nodes; the integration points required by the swarm client; and the security and liveness assumptions of the mechanism. The join procedure consists of two transactions, `register` and `activate`; everything between them — entropy finalization, target-prefix computation, and overlay mining — happens off-chain or through read-only calls. The registry holds no reservations: the target prefix is at all times a deterministic, read-only computation over current contract state. A target invalidated by an intervening activation simply yields a new target; under realistic churn such races are rare and cost only local mining work. +The mechanism is designed to achieve: + +- complete coverage of the address space by disjoint areas of responsibility; +- a maximum factor of two between the largest and smallest areas; +- uniformly random selection among currently eligible split candidates or donor pairs; +- assignment that is unknown to an applicant when it commits stake and fees; +- a bounded, explicit procedure for failed joins and failed donor relocations; +- logarithmic contract work in the number of active nodes for selection and tree updates; and +- deterministic validation of assignments by the contract and swarm node clients. + +The following are explicit non-goals: + +- proving that two Ethereum addresses belong to different operators; +- enforcing one human or one operator per node; +- preventing a well-funded operator from registering many independently assigned nodes; +- defining the postage redistribution game itself; +- defining the stake amount, application fee, reward, or slashing schedule; and +- specifying the transfer of stored content between nodes (see [Data handover](#data-handover)). + +Stake, fees, activation deadlines, and relocation deadlines are nevertheless security parameters. Their values MUST be specified by deployment configuration or by the staking specification on which a deployment depends. + ### Terminology **Address space** — the set $\Sigma^{256}$, where $\Sigma=\{0,1\}$. -**Overlay** — node overlay address derived from a node's public key, network ID, and overlay nonce using Bee's canonical overlay derivation function. +**Overlay** — node overlay address derived from a node's public key, network ID, and overlay nonce using Swarm's canonical overlay derivation function. **Prefix** — a bit string $p\in\Sigma^\ell$. Its length $\ell$ is its depth. @@ -42,7 +56,7 @@ The join procedure consists of two transactions, `register` and `activate`; ever **Leaf** — a prefix currently assigned to exactly one active node. Active leaves form a complete, prefix-free partition of the address space. -**Sibling** — the other child of the same parent prefix. For $p=b_0\ldots b_{\ell-1}$, its sibling is $b_0\ldots(1-b_{\ell-1})$. +**Sibling/sister** — the other child of the same parent prefix. For $p=b_0\ldots b_{\ell-1}$, its sibling is $b_0\ldots(1-b_{\ell-1})$. **Split candidate** — an active leaf at the current minimum depth $d$. A join splits this leaf into two child prefixes. @@ -56,29 +70,6 @@ The join procedure consists of two transactions, `register` and `activate`; ever **Active node count** — $N$, the number of activated nodes in the balanced partition. A pending registration is not active. -### Goals and explicit non-goals - -The goals of this SWIP are: - -- complete coverage of the address space by disjoint areas of responsibility; -- a maximum factor of two between the largest and smallest areas; -- uniformly random selection among currently eligible split candidates or donor pairs; -- assignment that is unknown to an applicant when it commits stake and fees; -- a bounded, explicit procedure for failed joins and failed donor relocations; -- logarithmic contract work in the number of active nodes for selection and tree updates; and -- deterministic validation of assignments by the contract and Bee clients. - -The following are explicit non-goals: - -- proving that two Ethereum addresses belong to different operators; -- enforcing one human or one operator per node; -- preventing a well-funded operator from registering many independently assigned nodes; -- defining the postage redistribution game itself; -- defining the stake amount, application fee, reward, or slashing schedule; and -- specifying the transfer of stored content between nodes (see [Data handover](#data-handover)). - -Stake, fees, activation deadlines, and relocation deadlines are nevertheless security parameters. Their values MUST be specified by deployment configuration or by the staking specification on which a deployment depends. - ## Motivation Swarm assigns work according to proximity in the overlay address space. A node is responsible for jobs, chunks, or protocol actions whose identifiers fall within an address range close to its overlay. If participating overlays cluster, some parts of the address space receive more providers than others, responsibility becomes uneven, and operators may deliberately co-locate identities. @@ -93,7 +84,7 @@ Fixed stake per active node is compatible with this construction because equally ## Solution -Let $S=\{n_0,\ldots,n_{N-1}\}$ be the set of active nodes. Node $n_i$ has an Ethereum staking identity $a_i$, a Bee public key $P_i$, and an overlay $o_i\in\Sigma^{256}$. The staking identity MUST authorize the public key used to derive the overlay. +Let $S=\{n_0,\ldots,n_{N-1}\}$ be the set of active nodes. Node $n_i$ has an Ethereum staking identity $a_i$, a Swarm node public key $P_i$, and an overlay $o_i\in\Sigma^{256}$. The staking identity MUST authorize the public key used to derive the overlay. For a prefix $p\in\Sigma^\ell$, define: @@ -172,15 +163,15 @@ The join protocol consists of exactly two transactions: `register` and `activate The applicant submits: - its staking identity $a_i$; -- its Bee public key $P_i$; +- its Swarm node public key $P_i$; - the required stake; - a non-refundable application fee. -The contract creates a registration with a unique request ID, records the registration block height $h_i$, and appends the entry to the registration queue. An identity MUST NOT have more than one live registration or active assignment. The stake remains locked until the registration activates or expires according to the staking rules. +The contract records the registration block height $h_i$ and appends the entry to the registration queue. An identity MUST NOT have more than one live registration or active assignment, so registrations are keyed by the staking identity itself — no separate request ID exists. The stake remains locked until the registration activates or expires according to the staking rules. Registering does not modify the active prefix tree. -The registration is valid for a _validity window_ of $VW$ blocks. In practice $VW < 256$, the number of blocks for which a block hash remains available from within the EVM. Since block heights in the registration queue are monotonically increasing, entries expire from the front; a permissionless `expire` call advances the queue head past expired entries and burns their fees, without ever iterating the full queue. +The registration is valid for a _validity window_ of $VW$ blocks. In practice $VW < 256$, the number of blocks for which a block hash remains available from within the EVM. Since block heights in the registration queue are monotonically increasing, entries expire from the front; a permissionless `expireReg` call advances the queue head past expired entries and burns their fees, without ever iterating the full queue. #### 2. Compute the target prefix @@ -192,7 +183,7 @@ For $N=0$, selection is omitted and the target is the empty prefix. There is no #### 3. Mine an overlay -The applicant mines a nonce $\nu_i$ such that Bee's canonical overlay derivation: +The applicant mines a nonce $\nu_i$ such that Swarm's canonical overlay derivation: $$ o_i=\operatorname{Overlay}(P_i,\operatorname{networkID},\nu_i) @@ -245,30 +236,31 @@ The donor MAY operate its old and new overlays concurrently during the transitio If the donor fails to activate within the validity window, it is dropped from the registry and its stake is forfeited; a fresh donor is drawn with new entropy and the procedure repeats. The departing node remains active until a donor completes. +Any number of departures MAY pend concurrently: each records its own donor and relocation deadline, and a stalled donor delays only its own departure. Concurrent draws never collide, because the donor's sibling takeover is applied to the tree at draw time, so every draw selects from the currently remaining pairs. A leaf whose departure is pending is excluded from split-candidate eligibility, so a join cannot split a neighbourhood that is about to be vacated. + #### Data handover This SWIP specifies only the assignment of nodes to neighbourhoods; the handover of stored content during relocation is deliberately left unspecified. A relocated node is expected to synchronise the reserve of its newly assigned neighbourhood, while the neighbourhood it vacated retains the data with the sibling node. With upcoming durability guarantees, such relocation gaps become repairable events; in particular, a newly assigned node need not sync its reserve live from its sibling but may instead acquire the neighbourhood's content from cold storage. ### Randomness and anti-grinding measures -For request $q$, registered in block $h_q$, let the entropy block be: +For a registration by identity $a$, committed in block $h_a$, let the entropy block be: $$ -t_q=h_q+\Delta, +t_a=h_a+\Delta, $$ -where $\Delta\geq1$ is a configured delay. The seed becomes computable only after $t_q$ has the configured number of confirmations, and remains computable only while its block hash is available to the contract. +where $\Delta\geq1$ is a configured delay. The seed becomes computable only after $t_a$ has the configured number of confirmations, and remains computable only while its block hash is available to the contract. The seed is domain-separated: $$ -\rho_q=\operatorname{keccak256}( +\rho_a=\operatorname{keccak256}( \texttt{"SWIP39"}\parallel \operatorname{chainID}\parallel \operatorname{contractAddress}\parallel -q\parallel -a_q\parallel -\operatorname{blockhash}(t_q)). +a\parallel +\operatorname{blockhash}(t_a)). $$ The deployment MUST specify $\Delta$, the confirmation delay, and the validity window. On an EVM deployment using the `BLOCKHASH` opcode, activation MUST occur before the referenced hash becomes unavailable. @@ -293,7 +285,7 @@ The following measures limit assignment grinding: - an identity has at most one live registration; - the application fee is not refunded after entropy becomes available; - an expired registration forfeits its fee; -- request IDs and identities are included in the seed; and +- the registering identity is included in the seed; and - selection is computed from contract state, not from a caller-supplied list. These measures price selective aborts but do not prevent one operator from funding multiple identities. Security analysis MUST therefore express resistance in economic and probabilistic terms rather than as a one-operator-one-node guarantee. @@ -307,38 +299,40 @@ The registry maintains: - `activeCount`: the number $N$ of active nodes; - `leaves`: trie leaves containing owner, public-key commitment, and overlay; - `participants`: staking identity to active assignment; -- `registrations`: request ID to registration record; -- `registrationQueue` ($C_R$) and `departureQueue` ($C_D$): pending request IDs in commit order; +- `registrations`: staking identity to registration record; +- `registrationQueue` ($C_R$) and `departureQueue` ($C_D$): pending identities in commit order; - subtree counts for split-candidate and donor-pair selection; -- `pendingDeparture`: the departure currently awaiting donor relocation, if any; and +- `pendingDepartures`: for each departure awaiting donor relocation (recorded in $C_D$), the drawn donor and its relocation deadline; and - configuration parameters and references to the staking contract. -Joins require no coordination state: registrations queue freely and activations validate against current state. Only donor relocation pends: +Joins require no coordination state: registrations queue freely and activations validate against current state. Only donor relocations pend, and any number of them may be in flight at once, each independent of the others. The lifecycle of any one departure: | State | Meaning | Permitted terminal action | |---|---|---| -| `Idle` | No pending rebalancing | Registrations, activations, and direct departures proceed freely | -| `DonorPending` | Donor drawn, relocation in progress | Donor activates and the departure completes, or the deadline passes: stake forfeited, fresh donor drawn | +| `Active` | No pending rebalancing | Registrations, activations, and direct departures proceed freely | +| `DonorPending` | Donor drawn, relocation in progress | Donor activates and the departure completes; the deadline passing moves to `Expired` | +| `Expired` | Donor missed its relocation deadline | `expireDereg` forfeits the donor's stake and draws a fresh donor | ```mermaid stateDiagram-v2 - [*] --> Idle - Idle --> Idle: register / activate / direct deregister - Idle --> DonorPending: deregister requiring rebalancing - DonorPending --> Idle: donor activates, departure completes - DonorPending --> DonorPending: deadline passed, stake forfeited, fresh donor drawn + [*] --> Active + Active --> Active: register / activate / direct deregister + Active --> DonorPending: deregister requiring rebalancing + DonorPending --> Active: donor activates, departure completes + DonorPending --> Expired: relocation deadline passes + Expired --> DonorPending: expireDereg — stake forfeited, fresh donor drawn ``` -Contract methods and Bee HTTP endpoints are distinct interfaces. A reference contract is expected to expose operations equivalent to: +Contract methods and swarm client HTTP endpoints are distinct interfaces. A reference contract is expected to expose operations equivalent to: - `register(publicKeyCommitment)`; -- `targetPrefix(identity)` (read-only); -- `activate(requestID, nonce, authorization)`; -- `expire()`; +- `target(identity)` (read-only): the current target prefix of a live registration; +- `activate(nonce, authorization)`; +- `expireReg()`: clean the registration queue; - `deregister()`; -- `expireDonor(leaveID)`; -- `getAssignment(identity)`; and -- `closestActiveNode(address)`. +- `expireDereg(identity)`: forfeit a defaulted donor and redraw; +- `getPrefix(identity)` (read-only): the assigned prefix of an active node; and +- `nodeFor(address)` (read-only): the active node closest to an address. All state-changing methods MUST validate the expected state and deadline. External calls to staking or proof-verification contracts MUST follow checks-effects-interactions or an equivalent reentrancy-safe pattern. @@ -443,7 +437,7 @@ selectDonor(r, s): # rank r over donor pairs, s one extra seed return leaf(j) # the moving donor ``` -The bit $s$ is taken from the same seed $\rho_q$ (e.g. the next bit after those consumed by rejection sampling), so the choice within the pair is as unpredictable as the pair itself. The donor's sibling $\operatorname{Sibling}(j)$ takes over the pair's parent prefix $i$, which is what makes the donor's removal a balanced one. +The bit $s$ is taken from the same seed $\rho_a$ (e.g. the next bit after those consumed by rejection sampling), so the choice within the pair is as unpredictable as the pair itself. The donor's sibling $\operatorname{Sibling}(j)$ takes over the pair's parent prefix $i$, which is what makes the donor's removal a balanced one. #### Commit queues and expiry @@ -460,9 +454,9 @@ expire(queue, maxSteps): steps = steps + 1 ``` -This is $O(1)$ amortized and bounded per call by `maxSteps`. It SHOULD be invoked at the start of `activate` and `deregister` processing, so the queues are clean before any state transition; after `expire`, checking that a registration is live reduces to verifying that its entry sits at or beyond the head. +This is $O(1)$ amortized and bounded per call by `maxSteps`. The public entry points `expireReg` and `expireDereg` apply it to $C_R$ and $C_D$ respectively, and it SHOULD also run at the start of `activate` and `deregister` processing, so the queues are clean before any state transition; afterwards, checking that a registration is live reduces to verifying that its entry sits at or beyond the head. -A donor's re-entry is pushed with the block height recorded by the `deregister` call, so it expires on the same schedule as the departure that caused it. A defaulted donor's entry is passed over like any expired entry — its stake forfeited by `expireDonor`, which pushes the fresh donor's entry at the current height. +A donor's re-entry is pushed with the block height recorded by the `deregister` call, so it expires on the same schedule as the departure that caused it. A defaulted donor's entry is passed over like any expired entry — its stake forfeited by `expireDereg`, which pushes the fresh donor's entry at the current height. For $N\geq1$, joining terminals are active leaves at depth $d$. Donor-selection terminals are depth-$d$ parents whose two children are active leaves. The contract MUST assert before selection that the root count equals the count implied by the invariant: @@ -480,7 +474,7 @@ These assertions provide useful invariant checks in tests even if production cod ### Client integration -Bee requires an operator-facing workflow for both initial assignment and relocation: +The swarm node client software requires an operator-facing workflow for both initial assignment and relocation: 1. create or load the node key; 2. register its public-key commitment through the staking/registry integration; @@ -491,11 +485,11 @@ Bee requires an operator-facing workflow for both initial assignment and relocat 7. start or update the node with the activated overlay; and 8. expose registration and relocation progress. -The contract and Bee MUST use exactly the same overlay derivation function, field encoding, network ID, and bit ordering. These values MUST be fixed by test vectors before deployment. +The contract and the swarm node client MUST use exactly the same overlay derivation function, field encoding, network ID, and bit ordering. These values MUST be fixed by test vectors before deployment. -For relocation, Bee SHOULD support a staging mode in which the old overlay continues serving while the new overlay is mined and synchronized. What data moves, and how, is outside this SWIP (see [Data handover](#data-handover)). +For relocation, the swarm node client SHOULD support a staging mode in which the old overlay continues serving while the new overlay is mined and synchronized. What data moves, and how, is outside this SWIP (see [Data handover](#data-handover)). -Read-only Bee endpoints MAY expose registry state, but they MUST NOT be confused with contract methods. Suggested client operations are: +Read-only client endpoints MAY expose registry state, but they MUST NOT be confused with contract methods. Suggested client operations are: - begin or inspect registration; - report the current target prefix and expiry; @@ -513,7 +507,7 @@ The registry improves distribution only under explicit identity, randomness, and The adversary may: - create and fund multiple Ethereum identities; -- operate multiple Bee nodes; +- operate multiple Swarm nodes; - abandon an assignment after learning its prefix; - reorder, front-run, or censor transactions when it controls block production; - attempt to bias a future block hash; @@ -789,4 +783,4 @@ and that every active overlay starts with its assigned prefix. The implementation SHOULD also generate random valid sequences of joins, direct departures, donor relocations, expiries, and redraws, comparing contract state against a simple off-chain model after each completed transition. -The exact Solidity ABI, Bee API paths, economic parameters, and deployment addresses remain to be supplied before this SWIP can advance beyond Draft. +The exact Solidity ABI, client API paths, economic parameters, and deployment addresses remain to be supplied before this SWIP can advance beyond Draft. From 03b159fc328cfc89734f8e1013c9325a1be66c5c Mon Sep 17 00:00:00 2001 From: zelig Date: Mon, 27 Jul 2026 03:30:41 +0200 Subject: [PATCH 23/23] replace \operatorname with \mathrm (macro not allowed by GitHub math renderer) Co-Authored-By: Claude Fable 5 --- SWIPs/swip-39.md | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/SWIPs/swip-39.md b/SWIPs/swip-39.md index 97ed394d..fd182db3 100644 --- a/SWIPs/swip-39.md +++ b/SWIPs/swip-39.md @@ -186,7 +186,7 @@ For $N=0$, selection is omitted and the target is the empty prefix. There is no The applicant mines a nonce $\nu_i$ such that Swarm's canonical overlay derivation: $$ -o_i=\operatorname{Overlay}(P_i,\operatorname{networkID},\nu_i) +o_i=\mathrm{Overlay}(P_i,\mathrm{networkID},\nu_i) $$ starts with the target prefix. @@ -255,12 +255,12 @@ where $\Delta\geq1$ is a configured delay. The seed becomes computable only afte The seed is domain-separated: $$ -\rho_a=\operatorname{keccak256}( +\rho_a=\mathrm{keccak256}( \texttt{"SWIP39"}\parallel -\operatorname{chainID}\parallel -\operatorname{contractAddress}\parallel +\mathrm{chainID}\parallel +\mathrm{contractAddress}\parallel a\parallel -\operatorname{blockhash}(t_a)). +\mathrm{blockhash}(t_a)). $$ The deployment MUST specify $\Delta$, the confirmation delay, and the validity window. On an EVM deployment using the `BLOCKHASH` opcode, activation MUST occur before the referenced hash becomes unavailable. @@ -346,15 +346,15 @@ An _implicit complete binary trie (ICBT)_ uses one-based heap indices; no pointe | Description | Notation | Definition | |---|---|---| -| depth of $i$ | $\operatorname{Depth}(i)$ | $\lfloor\log_2 i\rfloor$ | -| parent of $i$ | $\operatorname{Parent}(i)$ | $\lfloor i/2\rfloor$ | -| left child of $i$ | $\operatorname{Left}(i)$ | $2i$ | -| right child of $i$ | $\operatorname{Right}(i)$ | $2i+1$ | -| sibling of $i$ | $\operatorname{Sibling}(i)$ | $i+1-2(i\bmod 2)$ | -| position of $i$ within its level | $\operatorname{Position}(i)$ | $i-2^{\operatorname{Depth}(i)}$ | -| index of prefix $p$ with $\lvert p\rvert=\ell$ | $\operatorname{Index}(p)$ | $2^\ell+\operatorname{uint}(p)$ | +| depth of $i$ | $\mathrm{Depth}(i)$ | $\lfloor\log_2 i\rfloor$ | +| parent of $i$ | $\mathrm{Parent}(i)$ | $\lfloor i/2\rfloor$ | +| left child of $i$ | $\mathrm{Left}(i)$ | $2i$ | +| right child of $i$ | $\mathrm{Right}(i)$ | $2i+1$ | +| sibling of $i$ | $\mathrm{Sibling}(i)$ | $i+1-2(i\bmod 2)$ | +| position of $i$ within its level | $\mathrm{Position}(i)$ | $i-2^{\mathrm{Depth}(i)}$ | +| index of prefix $p$ with $\lvert p\rvert=\ell$ | $\mathrm{Index}(p)$ | $2^\ell+\mathrm{uint}(p)$ | -The binary representation of $\operatorname{Position}(i)$, padded to $\operatorname{Depth}(i)$ bits, is exactly the prefix designated by index $i$ — so the constraint prefix handed to the miner can be read off the index alone, with no storage access. The first three levels: +The binary representation of $\mathrm{Position}(i)$, padded to $\mathrm{Depth}(i)$ bits, is exactly the prefix designated by index $i$ — so the constraint prefix handed to the miner can be read off the index alone, with no storage access. The first three levels: ```mermaid graph TD @@ -421,7 +421,7 @@ targetPrefix(r): return (c << 1) | (1 - b) # applicant takes the sibling child ``` -Since the prefix equals the binary representation of $\operatorname{Position}(i)$, the accumulator `c` is redundant with the final index — it is shown for clarity; an implementation may just return $\operatorname{Position}$ of the selected child. +Since the prefix equals the binary representation of $\mathrm{Position}(i)$, the accumulator `c` is redundant with the final index — it is shown for clarity; an implementation may just return $\mathrm{Position}$ of the selected child. Donor selection descends the same way over `donorCount`, terminating at a depth-$d$ parent whose two children are active leaves, and then consumes one further random bit to pick the moving donor from the pair: @@ -437,7 +437,7 @@ selectDonor(r, s): # rank r over donor pairs, s one extra seed return leaf(j) # the moving donor ``` -The bit $s$ is taken from the same seed $\rho_a$ (e.g. the next bit after those consumed by rejection sampling), so the choice within the pair is as unpredictable as the pair itself. The donor's sibling $\operatorname{Sibling}(j)$ takes over the pair's parent prefix $i$, which is what makes the donor's removal a balanced one. +The bit $s$ is taken from the same seed $\rho_a$ (e.g. the next bit after those consumed by rejection sampling), so the choice within the pair is as unpredictable as the pair itself. The donor's sibling $\mathrm{Sibling}(j)$ takes over the pair's parent prefix $i$, which is what makes the donor's removal a balanced one. #### Commit queues and expiry @@ -461,11 +461,11 @@ A donor's re-entry is pushed with the block height recorded by the `deregister` For $N\geq1$, joining terminals are active leaves at depth $d$. Donor-selection terminals are depth-$d$ parents whose two children are active leaves. The contract MUST assert before selection that the root count equals the count implied by the invariant: $$ -\operatorname{splitCount}(1)=2^{d+1}-N, +\mathrm{splitCount}(1)=2^{d+1}-N, $$ $$ -\operatorname{donorCount}(1)=N-2^d. +\mathrm{donorCount}(1)=N-2^d. $$ For $N=0$, the implementation uses the empty prefix as a single bootstrap slot and has no donor pair. @@ -658,7 +658,7 @@ $$ Thus: $$ -\operatorname{splitCount}(1)=2^{3}-5=3. +\mathrm{splitCount}(1)=2^{3}-5=3. $$ Suppose uniform selection yields leaf $10$, and the incumbent overlay starts with $100$. The incumbent keeps child $100$, while the applicant targets $101$. After activation: