This is the second part of a [series](https://cowlicks.website/categories/hypercore.html) of posts about rewriting [Hypercore](https://github.com/holepunchto/hypercore) in [Rust](https://github.com/datrs/hypercore)
There is little guidance on how to test Rust against code from other languages.
After trying with several patterns, I found that the approach described here to be the most elegant.
I've been working on rewriting [`Hyperbee`](https://github.com/cowlicks/hyperbee) in Rust.
Hyperbee is a BTree built on Hypercore, a peer-to-peer append-only log, originally written in JavaScript.
I needed to be able to test my Rust code against the equivalent JavaScript.
I'm also creating a Python Hyperbee library, that wraps the Rust, and so I need to test Python against the equivalent Rust.
This is all part of the [Rust Hypercore](https://github.com/datrs) project which is working to rewrite all of Hypercore in Rust.
Another goal rewrite, is to be able to use Rust's excellent [Foreign Function Interface (FFI)](https://doc.rust-lang.org/nomicon/ffi.html) to create libraries for other languages, starting with Python.
So having a good pattern for cross language testing would prove useful throughout this project.
## Key Benefits
### No extra steps, just `cargo test`
Running your entire test suite through `cargo test` streamlines the testing process.
As long as you have the foreign language runtime and `make` installed, there's no additional setup needed.
This aligns with Rust developers' expectations - they're used to `cargo test` handling everything.
No need to to remember to run a separate command like`pytest` or `npm test`.
By avoiding these extra steps, we reduce friction and potential confusion in the development workflow.
### Collocated Test Code
Writing foreign language test code directly within your Rust tests offers several advantages:
* Test code lives alongside the Rust code it's testing
* Clear documentation of test purpose through proximity
* No need to manage hard coded values
* Easy to extend tests as both implementations evolve
* Reduced overhead from switching between files
This organization makes the tests more maintainable and self-documenting. When you need to understand how a piece of Rust code interacts with Python or JavaScript, everything you need is in one place.
## How it works
Lets start with an [example](https://github.com/cowlicks/hyperbee/tree/master/tests) directory structure taken from Hyperbeee.
```
# crate root
tests/ # Integration tests directory
py_tests.rs # Rust -> Python tests
js_tests.rs # Rust -> JavaScript tests
common/
Makefile # Makefile for non-language specific stuff
vars.mk # Variables shared across Makefiles
mod.rs # Rust module
...
python/ # Python specifics
Makefile # Run `make` to set up python dependencies
mod.rs # python specific rust code
... # misc python files like requirements.txt
javascript/ # JavaScript specifics
Makefile # Run `make` to set up js dependencies
mod.rs # javascript specific rust code
... # misc js files: package.json, node_module, etc
```
Here `*_tests.rs` are Rust tests. Each Rust test file is compiled as a crate, from those crates `common/` is seen as a Rust module.
We will explain the innards of `common` later, but first we will start by looking at a test.
## A Rust Test using Python
Let's dive into `py_test.rs` to see how this works
```rust
mod common;
use common::{
python::{run_python},
Result,
};
use hyperbee::Hyperbee;
#[tokio::test]
async fn hello_world() -> Result<()> {
let storage_dir = tempfile::tempdir()?;
{
let hb = Hyperbee::from_storage_dir(&storage_dir).await?;
hb.put(b"hello", Some(b"world")).await?;
// drop `hb` so it does not lock `storage_dir`
}
let result = run_python(&format!(
"
async def main():
hb = await hyperbee_from_storage_dir('{}')
res = await hb.get(b'hello')
assert(res.value == b'world')
",
storage_dir.path().display()
))?;
assert_eq!(result.status.code(), Some(0));
Ok(())
}
```
The actual test part is simple
The Rust creates a Hyperbee and inserts the key `b"hello"` with value `b"world"`, which is saved to `storage_dir`.
Then Python opens a Hyperbee with the same `storage_dir`, and checks for the key value.
The way the test works is more interesting.
At the top we declare `mod common` to make the `common` module available. We see this in the directory alongside `py_test.rs`. Within common we can have code exclusively for tests. We import `common::python::{require_python, run_python}`. These come from `tests/common/python/mod.rs`.
Rust code in the `commod` module is general purpose test code. Code within the `python` and `javascript` modules is language specific Rust code.
### Building test dependencies
To understand what's going on we step into `run_python`:
```rust
/// Run the provided python `code` within a context where the Python Hyperbee library is imported.
/// Code must be written within a `async main(): ...` function.
pub fn run_python(code: &str) -> Result> {
require_python()?;
run_code(
&build_whole_script(PRE_SCRIPT, code, POST_SCRIPT),
"script.py",
build_command,
vec![path_to_python_target()?, path_to_c_lib()?],
)
}
```
The first thing do is call `require_python`.
This ensures that all the dependencies for running the python code are ready.
It looks like:
```rust
pub fn require_python() -> Result> {
run_make_from_dir_with_arg(REL_PATH_TO_HERE, "")
}
```
It just calls `run_make_from_dir_with_arg`
```rust
pub fn run_make_from_dir_with_arg(dir: &str, arg: &str) -> Result