Self encrypting files (convergent encryption plus obfuscation)
| Crate | Documentation |
|---|---|
 |
 |
| Autonomi | Documentation | Discord |
|---|
A version of convergent encryption with an additional obfuscation step. This pattern allows secured data that can also be de-duplicated. This library presents an API that takes a set of bytes and returns a secret key derived from those bytes, and a set of encrypted chunks.
Important Security Note: While this library provides very secure encryption of the data, the returned secret key requires the same secure handling as would be necessary for any secret key.
- Content-based chunking
- Convergent encryption
- Self-validating chunks
- Hierarchical data maps for handling large files
- Streaming encryption/decryption
- Python bindings
- Flexible storage backend support
- Custom storage backends via functors
Add this to your Cargo.toml:
[dependencies]
self_encryption = "0.30"
bytes = "1.0"use self_encryption::{encrypt, decrypt_full_set};
use bytes::Bytes;
// Basic encryption/decryption
fn basic_example() -> Result<()> {
let data = Bytes::from("Hello, World!".repeat(1000)); // Must be at least 3072 bytes
// Encrypt data
let (data_map, encrypted_chunks) = encrypt(data.clone())?;
// Decrypt data
let decrypted = decrypt(&data_map, &encrypted_chunks)?;
assert_eq!(data, decrypted);
Ok(())
}use self_encryption::{shrink_data_map, get_root_data_map};
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
// Memory Storage Example
fn memory_storage_example() -> Result<()> {
let storage = Arc::new(Mutex::new(HashMap::new()));
// Store function
let store = |hash, data| {
storage.lock().unwrap().insert(hash, data);
Ok(())
};
// Retrieve function
let retrieve = |hash| {
storage.lock().unwrap()
.get(&hash)
.cloned()
.ok_or_else(|| Error::Generic("Chunk not found".into()))
};
// Use with data map operations
let shrunk_map = shrink_data_map(data_map, store)?;
let root_map = get_root_data_map(shrunk_map, retrieve)?;
Ok(())
}
// Disk Storage Example
fn disk_storage_example() -> Result<()> {
let chunk_dir = PathBuf::from("chunks");
// Store function
let store = |hash, data| {
let path = chunk_dir.join(hex::encode(hash));
std::fs::write(path, data)?;
Ok(())
};
// Retrieve function
let retrieve = |hash| {
let path = chunk_dir.join(hex::encode(hash));
Ok(Bytes::from(std::fs::read(path)?))
};
// Use with data map operations
let shrunk_map = shrink_data_map(data_map, store)?;
let root_map = get_root_data_map(shrunk_map, retrieve)?;
Ok(())
}pip install self-encryptionfrom self_encryption import encrypt, decrypt
# Basic in-memory encryption/decryption
def basic_example():
# Create test data (must be at least 3072 bytes)
data = b"Hello, World!" * 1000
# Encrypt data - returns data map and encrypted chunks
data_map, chunks = encrypt(data)
print(f"Data encrypted into {len(chunks)} chunks")
print(f"Data map has child level: {data_map.child()}")
# Decrypt data
decrypted = decrypt(data_map, chunks)
assert data == decrypted- Files are split into chunks of up to 1MB
- Each chunk is processed in three steps:
- Compression (using Brotli)
- Encryption (using AES-256-CBC)
- XOR obfuscation
-
Each chunk's encryption uses keys derived from the content hashes of three chunks:
For chunk N: - Uses hashes from chunks [N, N+1, N+2] - Combined hash = hash(N) || hash(N+1) || hash(N+2) - Split into: - Pad (first X bytes) - Key (next 16 bytes for AES-256) - IV (final 16 bytes) -
This creates a chain of dependencies where each chunk's encryption depends on its neighbors
-
Provides both convergent encryption and additional security through the interdependencies
-
Content Chunking:
- File is split into chunks of optimal size
- Each chunk's raw content is hashed (SHA3-256)
- These hashes become part of the DataMap
-
Per-Chunk Processing:
// For each chunk: 1. Compress data using Brotli 2. Generate key materials: - Combine three consecutive chunk hashes - Extract pad, key, and IV 3. Encrypt compressed data using AES-256-CBC 4. XOR encrypted data with pad for obfuscation
-
DataMap Creation:
- Stores both pre-encryption (src) and post-encryption (dst) hashes
- Maintains chunk ordering and size information
- Required for both encryption and decryption processes
-
Chunk Retrieval:
- Use DataMap to identify required chunks
- Retrieve chunks using dst_hash as identifier
-
Per-Chunk Processing:
// For each chunk: 1. Regenerate key materials using src_hashes from DataMap 2. Remove XOR obfuscation using pad 3. Decrypt using AES-256-CBC with key and IV 4. Decompress using Brotli
-
Chunk Reassembly:
- Chunks are processed in order specified by DataMap
- Reassembled into original file
-
Flexible backend support through trait-based design
-
Supports both memory and disk-based storage
-
Streaming operations for memory efficiency
-
Hierarchical data maps for large files:
// DataMap shrinking for large files 1. Serialize large DataMap 2. Encrypt serialized map using same process 3. Create new DataMap with fewer chunks 4. Repeat until manageable size reached
- Content-based convergent encryption
- Additional security through chunk interdependencies
- Self-validating chunks through hash verification
- No single point of failure in chunk storage
- Tamper-evident through hash chains
- Parallel chunk processing where possible
- Streaming support for large files
- Efficient memory usage through chunking
- Optimized compression settings
- Configurable chunk sizes
This implementation provides a balance of:
- Security (through multiple encryption layers)
- Deduplication (through convergent encryption)
- Performance (through parallelization and streaming)
- Flexibility (through modular storage backends)
Licensed under either of
- MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)
- Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.
Want to contribute? Great 🎉
There are many ways to give back to the project, whether it be writing new code, fixing bugs, or just reporting errors. All forms of contributions are encouraged!
To prepare a new release:
-
Create a PR with version bump and changelog:
- Update version in
Cargo.tomlbased on Semantic Versioning - Add new version entry to
CHANGELOG.mdwith release date and changes - Example: PR #416
- Update version in
-
Run the release workflow manually:
- After PR is merged, go to GitHub Actions
- Click "Run workflow" to trigger the automated release process