Unit 1.1 - Introduction
This is the open-source Rust training material we use for our professional multi-day (embedded) Rust training programs.
We have been developing this material in the open since 2023 and have used it to deliver over 15 customized in-company training sessions since then.
The material comprises several modules based on the material of the teach-rs project (now part of Trifecta Tech Foundation) that we previously co-developed and that we contribute back to when improvements are made.
Slides and exercises
All of our training programs consists of an interactive mix of theory and guided exercises.
The theory is taught by our trainers using the slides which are linked at the top of most modules - see the top of this page as well.
We regularly update both slides and exercises, integrating the feedback we receive.
Improvements
If you spot any mistakes or have a suggestion, please open an issue or pull request on the GitHub repository.
License
Published under the CC-BY-SA 4.0 license, you are free to use and share this material as long as you give us appropriate credit and share any modifications you make under that same license.
Let us train you!
Studying on your own is good, but being guided through an interactive tailor-made program with your team is more productive, and more fun!
For all testimonials, an introduction of our expert trainers, and more information about the programs we offer, please refer to our professional training page.
Unit 1.2 - Introduction
Exercise 1.2.1: Setup Your Installation
In this file you'll find instructions on how to install the tools we'll use during the course.
All of these tools are available for Linux, macOS and Windows users. We'll need the tools to write and compile our Rust code, and allow for remote mentoring. Important: these instructions are to be followed before the start of the first tutorial. If you have any problems with installation, please contact the trainers ahead of time. We would prefer not to have to be address installation problems during the first session.
Rust and Cargo
First we'll need rustc, the standard Rust compiler.
rustc is generally not invoked directly, but through cargo, the Rust package manager.
rustup takes care of installing rustc and cargo.
This part is easy: go to https://rustup.rs and follow the instructions. Please make sure you're installing the latest default toolchain. Once done, run
rustc -V && cargo -V
The output should be something like this:
rustc 1.79.0 (129f3b996 2024-06-10)
cargo 1.79.0 (ffa9cf99a 2024-06-03)
Using Rustup, you can install Rust toolchains and components. More info:
Rustfmt and Clippy
To avoid discussions, Rust provides its own formatting tool, Rustfmt. We'll also be using Clippy, a collection of lints to analyze your code, that catches common mistakes for you. You'll find that Rusts Clippy can be a very helpful companion. Both Rustfmt and Clippy are installed by Rustup by default.
To run Rustfmt on your project, execute:
cargo fmt
To run clippy:
cargo clippy
More info:
Visual Studio Code
During the course, we will use Visual Studio Code (vscode) to write code in. Of course, you're free to use your favorite editor, but if you encounter problems, you can't rely on support from us. Also, we'll use VSCode to allow for remote collaboration and mentoring during remote training sessions.
You can find the installation instructions here: https://code.visualstudio.com/.
We will install some plugins as well. The first one is Rust-Analyzer. Installation instructions can be found here https://marketplace.visualstudio.com/items?itemName=rust-lang.rust-analyzer. Rust-Analyzer provides a lot of help during development and in indispensable when getting started with Rust.
Another plugin we'll use is CodeLLDB. This plugin enables debugging Rust code from within vscode. You can find instructions here: https://marketplace.visualstudio.com/items?itemName=vadimcn.vscode-lldb.
If you're following the training remotely, install the Live Share plugin as well. We will use the plugin to share code and provide help during remote training sessions. Installation instructions can be found here: https://marketplace.visualstudio.com/items?itemName=MS-vsliveshare.vsliveshare
More info:
Tip
This repo contains quite a lot of rust projects and due to the complicated setup of the repo Rust Analyzer can't autodiscover them well.
To fix this we've specified the projects manually in the .vscode/settings.json file. To reduce the burden on your computer, you can comment out any of the projects that we're not using in our training.
Git
During the training, you'll need the Git version control tool. If you haven't installed Git already, you can find instructions here: https://git-scm.com/book/en/v2/Getting-Started-Installing-Git. If you're new to Git, you'll also appreciate GitHubs intro to Git https://docs.github.com/en/get-started/using-git/about-git and the Git intro with vscode, which you can find here: https://www.youtube.com/watch?v=i_23KUAEtUM.
More info: https://www.youtube.com/playlist?list=PLg7s6cbtAD15G8lNyoaYDuKZSKyJrgwB-
Course code
Now that everything is installed, you can clone the source code repository using Git. The repository can be found here: https://github.com/tweedegolf/rust-training.
Instructions on cloning the repository can be found here: https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls
Trying it out
Now that you've got the code on your machine, navigate to it using your favorite terminal and run:
cd exercises/1-course-introduction/1-introduction/1-setup-your-installation
cargo run
This command may take a while to run the first time, as Cargo will first fetch the crate index from the registry.
It will compile and run the intro package, which you can find in exercises/1-course-introduction/1-introduction/1-setup-your-installation.
If everything goes well, you should see some output:
Compiling intro v0.1.0 ([/path/to/rust-workshop]/exercises/1-course-introduction/1-introduction/1-setup-your-installation)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.15s
Running `target/debug/intro`
🦀 Hello, world! 🦀
You've successfully compiled and run your first Rust project!
X: 2; Y: 2
If Rust-Analyzer is set up correctly, you can also click the '▶️ Run'-button that is shown in exercises/1-course-introduction/1-introduction/1-setup-your-installation/src/main.rs.
With CodeLLDB installed correctly, you can also start a debug session by clicking 'Debug', right next to the '▶️ Run'-button.
Play a little with setting breakpoints by clicking on a line number, making a red circle appear and stepping over/into/out of functions using the controls.
You can view variable values by hovering over them while execution is paused, or by expanding the 'Local' view under 'Variables' in the left panel during a debug session.
Instructions for FFI module
This section is relevant only if you're taking part in one of the modules on Rust FFI.
For doing FFI we will need to compile some C code and for that we need a C compiler installed.
We've chosen to use clang in our excercises.
The prerequisite is that calling clang in your terminal should work. If it doesn't, follow the instructions for your platform below.
Linux
For the bookworms using a Debian-like:
sudo apt update
sudo apt install clang
If you're on Arch, btw:
sudo pacman -S clang
For those tipping their Fedora's:
sudo dnf install clang
Windows
Always make sure to select the option to add the install to path.
Using winget:
winget install -i -e --id LLVM.LLVM
For the sweethearts using chocolatey:
choco install llvm
For the handsome people preferring manual installation:
- Go to the releases page: https://github.com/llvm/llvm-project/releases
- Go to a recent release
- Search for LLVM-[VERSION]-win64.exe and download it
- Run the exe
MacOS
Using brew:
brew install llvm
Then also add to path, e.g.:
echo 'export PATH="$(brew --prefix llvm)/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc
Instructions for embedded
This section is relevant only if you're taking part in one of the modules on embedded Rust.
Hardware
We will use the BBC micro:bit V2 and either you've already got it or we will bring it with us.
You'll also need a Micro-USB cable, but we're sure you've got one to spare.
Please check that everything is complete. If not, please contact us.
Software
Then, we'll install some tools needed to flash the mcu and inspect the code.
Install the thumbv7em-none-eabihf toolchain with the following command:
rustup target add thumbv7em-none-eabihf
We'll also install a couple of tools that let us inspect our binaries:
rustup component add llvm-tools
cargo install cargo-binutils
Now, let's install probe-rs. Follow the installation instructions. Probe-rs talks with the debug interface on the micro:bit, to flash your application, log messages, or even set breakpoints and read out device memory.
For Linux this is:
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/probe-rs/probe-rs/releases/latest/download/probe-rs-tools-installer.sh | sh
If you're on linux, you'll need to update your udev rules.
On ubuntu or fedora, run the following inside the workshop folder you just cloned;
sudo cp 99-microbit-v2.rules /etc/udev/rules.d
sudo udevadm control --reload-rules
sudo udevadm trigger
It's possible that probe-rs detects two debugging interfaces. This is known to happen on Windows.
In that case, go to the .cargo/config.toml and change the runner to the one that specifies the exact probe
it needs to use. Make sure the values are the same as what probe-rs reports is one of the interfaces.
Unsure? You can run probe-rs list to get a list of all connected probes.
Trying it out
Before we begin, we need to test our hardware. We'll be testing the nRF52833 microcontroller and the LSM303AGR motion sensor, that are present on the micro:bit V2. Make sure you have checked out the latest version of the workshop source.
Running the test
To test the hardware, please connect the micro:bit V2 to your pc, switch it on, and run
cd ./exercises/1-course-introduction/1-introduction/2-embedded
cargo run --release
If everything works correctly, you should now see the accelerometer samples being printed on the display. If not, don't worry and contact us.
Docs
Datasheets, manuals, and schematics of the parts we are using in the embedded workshops.
BBC micro:bit V2
nRF52833
LSM303AGR
Unit 2.1 - Basic Syntax
Exercise 2.1.1: Basic Syntax
Open exercises/2-foundations-of-rust/1-basic-syntax/1-basic-syntax in your editor. This folder contains a number of exercises with which you can practise basic Rust syntax.
While inside the exercises/2-foundations-of-rust/1-basic-syntax/1-basic-syntax folder, to get started, run:
cargo run --bin 01
This will try to compile exercise 1. Try and get the example to run, and continue on with the next exercise by replacing the number of the exercise in the cargo run command.
Some exercises contain unit tests. To run the test in src/bin/01.rs, run
cargo test --bin 01
Make sure all tests pass!
Unit 2.2 - Ownership and References
Exercise 2.2.1: Move Semantics
This exercise is adapted from the move semantics exercise from Rustlings
While inside the exercises/2-foundations-of-rust/2-ownership-and-references/1-move-semantics folder, to get started, run:
cargo run --bin 01
This will try to compile exercise 1. Try and get the example to run, and continue on with the next exercise by replacing the number of the exercise in the cargo run command.
Some exercises contain unit tests. To run the test in src/bin/01.rs, run
cargo test --bin 01
Make sure all tests pass!
01.rs should compile as is, but you'll have to make sure the others compile as well. For some exercises, instructions are included as doc comments at the top of the file. Make sure to adhere to them.
Exercise 2.2.2: Borrowing
Fix the two examples in the exercises/2-foundations-of-rust/2-ownership-and-references/2-borrowing crate! Don't forget you
can run individual binaries by using cargo run --bin 01 in that directory!
Make sure to follow the instructions that are in the comments!
Unit 2.3 - Advanced Syntax
Exercise 2.3.1: Error propagation
Follow the instructions in the comments of exercises/2-foundations-of-rust/3-advanced-syntax/1-error-propagation/src/main.rs!
Exercise 2.3.2: Error handling
Follow the instructions in the comments of exercises/2-foundations-of-rust/3-advanced-syntax/2-error-handling/src/main.rs!
Exercise 2.3.3: Slices
Follow the instructions in the comments of exercises/2-foundations-of-rust/3-advanced-syntax/3-slices/src/main.rs!
Don't take too much time on the extra assignment, instead come back later once
you've done the rest of the exercises.
Exercise 2.3.4: Ring Buffer
This is a bonus exercise! Follow the instructions in the comments of
exercises/2-foundations-of-rust/3-advanced-syntax/4-ring-buffer/src/main.rs!
Exercise 2.3.5: Boxed Data
Follow the instructions in the comments of exercises/2-foundations-of-rust/3-advanced-syntax/5-boxed-data/src/main.rs!
Unit 2.4 - Traits and Generics
Exercise 2.4.1: Vector math
In this exercise, we'll implement some basic vector math operations for our 2-dimensional vector type Vec2D.
2.4.1.A Vector addition
Let's start by implementing vector addition for Vec2D as Rust's std::ops::Add trait. Replace the todo!() such that it returns a new Vec2D of which the x component is the sum of the x components of the two input vectors, and similarly for the y component. If implemented correctly, the integer_addition test should now pass!
2.4.1.B Dot product
Now, let's implement the dot product for our vector type. Multiplying two vectors with a dot product should return a singular value, which is the sum of the products of the components. For example, [1, 2] * [3, 4] = 1 * 3 + 2 * 4 = 11. Implement this by adding a new impl block that implements the std::ops::Mul trait for Vec2D. Uncomment the integer_dot_product test to test your code!
2.4.1.C Making Vec2D generic
Currently, the x and y components of our Vec2D can only be 32-bit integers. What if we want to use other number types, such as floating-point numbers? Let's make Vec2D generic over a type T by changing the definition of Vec2D such that the x and y are generic.
We will also need to update our addition and dot product implementations to support this generic type T. Instead of implementing the add and mul functions for every type of number separately, we can leave the implementations generic by implementing them for any T that can be added/multiplied (i.e. any type T that has the Add and/or Mul trait). This means we can just change the start of the impl block, without needing to change the actual function implementations!
Hint 1 (generic implementations with trait bounds)
You can add a generic type to an `impl` block by writing `implHint 2 (the `Add` and `Mul` traits)
The `Add` and `Mul` traits have a generic `Output` type. This allowed us to implement the dot product as the `Mul` trait of `Vec2D` by setting the output to be a number. For our generic implementation, we want the generic type `T` to be something we can add and/or multiply. To specify this, we add `Add` and/or `Mul` trait bounds to `T`, but we also need to specify what output we expect from adding or multiplying two `T` values by writing e.g. `T: AddUncomment the float_addition and float_dot_product tests to check if our Vec2D now works with floating-point values!
Exercise 2.4.2: Local Storage Vec
This is a bonus exercise! We'll create a type called LocalStorageVec, which is generic list of items that resides either on the stack or the heap, depending on its size. If its size is small enough for items to be put on the stack, the LocalStorageVec buffer is backed by an array. LocalStorageVec is not only generic over the type (T) of items in the list, but also by the size (N) of this stack-located array using a relatively new feature called "const generics". Once the LocalStorageVec contains more items than fit in the array, a heap based Vec is allocated as space for the items to reside in.
Within this exercise, the objectives are annotated with a number of stars (⭐), indicating the difficulty. You are likely not to be able to finish all exercises during the tutorial session
Questions
- When is such a data structure more efficient than a standard
Vec? - What are the downsides, compared to just using a
Vec?
Open the exercises/2-foundations-of-rust/4-traits-and-generics/1-local-storage-vec crate. It contains a src/lib.rs file, meaning this crate is a library. lib.rs contains a number of tests, which can be run by calling cargo test. Don't worry if they don't pass or even compile right now: it's your job to fix that in this exercise. Most of the tests are commented out right now, to enable a step-by-step approach. Before you begin, have a look at the code and the comments in there, they contain various helpful clues.
2.4.2.A Defining the type ⭐
Currently, the LocalStorageVec enum is incomplete. Give it two variants: Stack and Heap. Stack contains two named fields, buf and len. buf will be the array with a capacity to hold N items of type T; len is a field of type usize that will denote the amount of items actually stored. The Heap variant has an unnamed field containing a Vec<T>. If you've defined the LocalStorageVec variants correctly, running cargo test should output something like
running 1 test
test test::it_compiles ... ignored, This test is just to validate the definition of `LocalStorageVec`. If it compiles, all is OK
test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s
This test does (and should) not run, but is just there for checking your variant definition.
Hint 1
You may be able to reverse-engineer the `LocalStorageVec` definition using the code of the `it_compiles` test case.Hint 2 (If you got stuck, but try to resist me for a while)
Below definition works. Read the code comments and make sure you understand what's going on.
#![allow(unused)] fn main() { // Define an enum `LocalStorageVec` that is generic over // type `T` and a constant `N` of type `usize` pub enum LocalStorageVec<T, const N: usize> { // Define a struct-like variant called `Stack` containing two named fields: // - `buf` is an array with elements of `T` of size `N` // - `len` is a field of type `usize` Stack { buf: [T; N], len: usize }, // Define a tuple-like variant called `Heap`, containing a single field // of type `Vec<T>`, which is a heap-based growable, contiguous list of `T` Heap(Vec<T>), } }
2.4.2.B impl-ing From<Vec<T>> ⭐
Uncomment the test it_from_vecs, and add an implementation for From<Vec<T>> to LocalStorageVec<T>. To do so, copy the following code in your lib.rs file and replace the todo! macro invocation with your code that creates a heap-based LocalStorageVec containing the passed Vec<T>.
#![allow(unused)] fn main() { impl<T, const N: usize> From<Vec<T>> for LocalStorageVec<T, N> { fn from(v: Vec<T>) -> Self { todo!("Implement me"); } } }
Question
- How would you pronounce the first line of the code you just copied in English?*
Run cargo test to validate your implementation.
2.4.2.C AsRef and AsMut ⭐⭐
AsRef and AsMut are used to implement cheap reference-to-reference coercion. For instance, our LocalStorageVec<T, N> is somewhat similar to a slice &[T], as both represent a contiguous series of T values. This is true whether the LocalStorageVec buffer resides on the stack or on the heap.
Uncomment the it_as_refs test case and implement AsRef<[T]> and AsMut<[T]>.
Hint
Make sure to take into account the value of `len` for the `Stack` variant of `LocalStorageVec` when creating a slice.2.4.2.D impl LocalStorageVec ⭐⭐
To make the LocalStorageVec more useful, we'll add more methods to it.
Create an impl-block for LocalStorageVec.
Don't forget to declare and provide the generic parameters.
For now, to make implementations easier, we will add a bound T, requiring that it implements Copy and Default.
First off, uncomment the test called it_constructs.
Make it compile and pass by creating a associated function called new on LocalStorageVec that creates a new, empty LocalStorageVec instance without heap allocation.
The next methods we'll implement are len, push, pop, insert, remove and clear:
lenreturns the length of theLocalStorageVecpushappends an item to the end of theLocalStorageVecand increments its length. Possibly moves the contents to the heap if they no longer fit on the stack.popremoves an item from the end of theLocalStorageVec, optionally returns it and decrements its length. If the length is 0,popreturnsNoneinsertinserts an item at the given index and increments the length of theLocalStorageVecremoveremoves an item at the given index and returns it.clearresets the length of theLocalStorageVecto 0.
Uncomment the corresponding test cases and make them compile and pass. Be sure to have a look at the methods provided for slices [T] and Vec<T> Specifically, [T]::copy_within and Vec::extend_from_slice can be of use.
2.4.2.E Iterator and IntoIterator ⭐⭐
Our LocalStorageVec can be used in the real world now, but we still shouldn't be satisfied. There are various traits in the standard library that we can implement for our LocalStorageVec that would make users of our crate happy.
First off, we will implement the IntoIterator and Iterator traits. Go ahead and uncomment the it_iters test case. Let's define a new type:
#![allow(unused)] fn main() { pub struct LocalStorageVecIter<T, const N: usize> { vec: LocalStorageVec<T, N>, counter: usize, } }
This is the type we'll implement the Iterator trait on. You'll need to specify the item this Iterator implementation yields, as well as an implementation for Iterator::next, which yields the next item. You'll be able to make this easier by bounding T to Default when implementing the Iterator trait, as then you can use the std::mem::take function to take an item from the LocalStorageVec and replace it with the default value for T.
Take a look at the list of methods under the 'provided methods' section. In there, lots of useful methods that come free with the implementation of the Iterator trait are defined, and implemented in terms of the next method. Knowing in the back of your head what methods there are, greatly helps in improving your efficiency in programming with Rust. Which of the provided methods can you override in order to make the implementation of LocalStorageVecIter more efficient, given that we can access the fields and methods of LocalStorageVec?
Now to instantiate a LocalStorageVecIter, implement the [IntoIter] trait for it, in such a way that calling into_iter yields a LocalStorageVecIter.
2.4.2.F Index ⭐⭐
To allow users of the LocalStorageVec to read items or slices from its buffer, we can implement the Index trait. This trait is generic over the type of the item used for indexing. In order to make our LocalStorageVec versatile, we should implement:
Index<usize>, allowing us to get a single item by callingvec[1];Index<RangeTo<usize>>, allowing us to get the firstnitems (excluding itemn) by callingvec[..n];Index<RangeFrom<usize>>, allowing us to get the lastnitems by callingvec[n..];Index<Range<usize>>, allowing us to get the items betweennandmitems (excluding itemm) by callingvec[n..m];
Each of these implementations can be implemented in terms of the as_ref implementation, as slices [T] all support indexing by the previous types. That is, [T] also implements Index for those types. Uncomment the it_indexes test case and run cargo test in order to validate your implementation.
2.4.2.G Removing bounds ⭐⭐
When we implemented the borrowing Iterator, we saw that it's possible to define methods in separate impl blocks with different type bounds. Some of the functionality you wrote used the assumption that T is both Copy and Default. However, this means that each of those methods are only defined for LocalStorageVecs containing items of type T that in fact do implement Copy and Default, which is not ideal. How many methods can you rewrite having one or both of these bounds removed?
2.4.2.H Borrowing Iterator ⭐⭐⭐
We've already got an iterator for LocalStorageVec, though it has the limitation that in order to construct it, the LocalStorageVec needs to be consumed. What if we only want to iterate over the items, and not consume them? We will need another iterator type, one that contains an immutable reference to the LocalStorageVec and that will thus need a lifetime annotation. Add a method called iter to LocalStorageVec that takes a shared &self reference, and instantiates the borrowing iterator. Implement the Iterator trait with the appropriate Item reference type for your borrowing iterator. To validate your code, uncomment and run the it_borrowing_iters test case.
Note that this time, the test won't compile if you require the items of LocalStorageVec be Copy! That means you'll have to define LocalStorageVec::iter in a new impl block that does not put this bound on T:
#![allow(unused)] fn main() { impl<T: Default + Copy, const N: usize> LocalStorageVec<T, N> { // Methods you've implemented so far } impl<T: const N: usize> LocalStorageVec<T, N> { pub fn iter(&self) -> /* TODO */ } }
Defining methods in separate impl blocks means some methods are not available for certain instances of the generic type. In our case, the new method is only available for LocalStorageVecs containing items of type T that implement both Copy and Default, but iter is available for all LocalStorageVecs.
2.4.2.I Generic Index ⭐⭐⭐⭐
You've probably duplicated a lot of code in exercise 2.4.2.F. We can reduce the boilerplate by defining an empty trait:
#![allow(unused)] fn main() { trait LocalStorageVecIndex {} }
First, implement this trait for usize, RangeTo<usize>, RangeFrom<usize>, and Range<usize>.
Next, replace the multiple implementations of Index with a single implementation. In English:
"For each type T, I and constant N of type usize,
implement Index<I> for LocalStorageVec<T, N>,
where I implements LocalStorageVecIndex
and [T] implements Index<I>"
If you've done this correctly, it_indexes should again compile and pass.
2.4.2.J Deref and DerefMut ⭐⭐⭐⭐
The next trait that makes our LocalStorageVec more flexible in use are Deref and DerefMut that utilize the 'deref coercion' feature of Rust to allow types to be treated as if they were some type they look like.
That would allow us to use any method that is defined on [T] by calling them on a LocalStorageVec.
Before continuing, read the section 'Treating a Type Like a Reference by Implementing the Deref Trait' from The Rust Programming Language (TRPL).
Don't confuse deref coercion with any kind of inheritance! Using Deref and DerefMut for inheritance is frowned upon in Rust.
Below, an implementation of Deref and DerefMut is provided in terms of the AsRef and AsMut implementations. Notice the specific way in which as_ref and as_mut are called.
#![allow(unused)] fn main() { impl<T, const N: usize> Deref for LocalStorageVec<T, N> { type Target = [T]; fn deref(&self) -> &Self::Target { <Self as AsRef<[T]>>::as_ref(self) } } impl<T, const N: usize> DerefMut for LocalStorageVec<T, N> { fn deref_mut(&mut self) -> &mut Self::Target { <Self as AsMut<[T]>>::as_mut(self) } } }
Question
- Replacing the implementation of
derefwithself.as_ref()results in a stack overflow when running an unoptimized version. Why? (Hint: deref coercion)
Unit 2.5 - Closures and Dynamic dispatch
Exercise 2.5.1: Config Reader
In this exercise, you'll work with dynamic dispatch to deserialize with serde_json or serde_yaml, depending on the file extension. The starter code is in exercises/2-foundations-of-rust/5-closures-and-dynamic-dispatch/1-config-reader. Fix the todo's in there.
To run the program, you'll need to pass the file to deserialize to the binary, not to Cargo. To do this, run
cargo run -- <FILE_PATH>
Deserializing both config.json and config.yml should result in the Config being printed correctly.
Unit 3.1 - Crate Engineering
Exercise 3.1.1: My Serde App
This exercise is adapted from the serde_lifetimes exercise by Ferrous Systems
Open exercises/3-crate-engineering/1-crate-engineering/1-my-serde-app/src/main.rs. In there, you'll find some Rust code we will do this exercise with.
We used todo!() macros to mark places where you should put code to make the program run. Look at the serde_json api for help.
Hint
Serde comes with two traits: `Serializable` and `Deserializable`. These traits can be `derive` d for your `struct` or `enum` types. Other `serde-*` crates use these traits to convert our data type from and to corresponding representation (`serde-json` to JSON, `serde-yaml` to YAML, etc.).How come
mainreturns ananyhow::Result<()>? By havingmainreturn a result, we can bubble errors up all the way to runtime. You can find more information about it in Rust By Example. Theanyhow::Resultis a more flexible type ofResult, which allows for easy conversion of error types.
What is that
r#"...thing?rin front of a string literal means it's a "raw" string. Escape sequences (\n,\", etc.) don't work, and thus they are very convenient for things like regular expressions, JSON literals, etc.Optionally
rcan be followed by one or more symbols (like#in our case), and then your string ends when there's a closing double quote followed by the same number of the same symbols. This is great for cases when you want to have double quotes inside your string literal. For our exampler#" ... "#works great for JSON. In rare cases you'd want to put two or more pound signs. Like, when you store CSS color values in your JSON strings:
#![allow(unused)] fn main() { // here `"#` would not terminate the string r##" { "color": "#ff00ff" } "## }
Exercise 3.1.2: Quizzer
In this exercise, you will create a Rust crate that adheres to the guidelines that were pointed out during the lecture. Additionally, you will add and use dependencies, create unit tests, and create some documentation. You can view this exercise as a stepping stone to the final project.
This exercise should be done in groups of 2 people
3.1.2.A Setting up ⭐
Create a new project using cargo new --name quizzer. Make sure it acts as both a binary and a library. That means there will be both a src/lib.rs and a src/bin/quizzer/main.rs file in your crate, where quizzer is the name of the binary:
$ tree
.
├── Cargo.toml
├── quiz.json
└── src
├── bin
│ └── quizzer
│ └── main.rs
└── lib.rs
Add the following dependencies to your Cargo.toml file. Below items contain links to their page on docs.rs. Make sure you get a general idea of what these crates are for and how they can be used. Don't dive too deep just yet.
anyhow1.0clap4.0 Also, skim over https://docs.rs/clap/latest/clap/_derive/_tutorial/index.htmlserde-json1.0serde1.0
Your Cargo.toml should look like this:
[package]
name = "quizzer"
version = "0.1.0"
edition = "2021"
### See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
anyhow = "1.0.66"
clap = { version = "4.0.18", features = ["derive"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0.87"
For clap and serde, the non-standard derive feature of each these crates is enabled. For clap, it allows us to derive the Parser trait, which greatly simplifies creating a CLI. The derive feaure from serde allows us to derive the Serialize and Deserialize traits on any struct we wish to serialize or deserialize using serde and its backends, in our case serde_json.
3.1.2.B Quizzer ⭐⭐⭐
This exercise is about both design and finding information. You'll have to figure out a model to represent your quiz questions, as well as a means to store them into a JSON file, and load them yourself. Also, you will have to find out how to parse the program arguments.
We will use the project we just set up to write a quiz game creator and player. You may add other dependencies as needed. It has the following functional requirements:
- It runs as a command-line tool in your terminal.
- It has two modes: question-entering mode and quiz mode. The mode is selected with a subcommand, passed as the first argument to the program.
- Question-entering mode: Allows for entering multiple-choice quiz questions, with 4 possible answers each, exactly 1 of them being correct. The questions are stored on disk as a JSON file.
- Quiz mode: Loads stored questions from the JSON file, presents the questions one-by-one to the player, reads and verifies the player input, and presents the score at the end of the game.
- Errors are correctly handled, i.e. your application does not panic if it encounters any unexpected situation. Use
anywhowand the question-mark (?) operator to make error-bubbling concise. You can read about the?-operator here: https://doc.rust-lang.org/reference/expressions/operator-expr.html#the-question-mark-operator - Logic concerning creating, storing, and loading quiz questions is defined in the library part of your crate.
- Functionality regarding user input (arg parsing, reading from stdin) is defined in the application code, not in your library.
- Logical units of your crate are divided up into modules.
Before you start coding, make sure you've listed all open questions and found answers to them. You're also encouraged to draw a simple diagram of the module structure of your application, annotating each module with its responsibilities.
Exercise 3.1.3: BSN
The BSN (Burgerservicennummer) is a Dutch personal identification number that somewhat resembles the US Social Security Number in its use. The BSN is a number that adheres to some rules. In this exercise, we will create a Rust type that guarantees that it represents a valid BSN.
3.1.3.A Newtype ⭐⭐
In this part we will implement the BSN number validation, as well as a fallible constructor.
A BSN is valid if and only if it matches the following criteria:
- It consists of 8 or 9 digits
- It passes a variant of the 11 check (elfproef (Dutch)):
For 8-digit BSNs, we concatenate a 0 to the end. The digits of the number are labeled as ABCDEFGHI.
For example: for BSN 123456789, A = 1, B = 2, C = 3, and so forth until I.
Then, (9 × A) + (8 × B) + (7 × C) + (6 × D) + (5 × E) + (4 × F) + (3 × G) + (2 × H) + (-1 × I) must be a multiple of 11
Open exercises/3-crate-engineering/1-crate-engineering/3-bsn in your editor. You'll find the scaffolding code there, along with two files:
valid_bsns.incontaining a list of valid BSNsinvalid_bsns.incontaining a list of invalid BSNs.
In src/lib.rs, implement Bsn::validate to make the test_validation test case pass.
Implement Bsn::try_from_string as well.
To try just the test_validation test case, run:
cargo test -- test_validation
3.1.3.B Visitor with Serde ⭐⭐⭐
Next up is implementing the serde::Serialize and serde::Deserialize traits, to support serialization and deserialization of Bsns.
In this case, simply deriving those traits won't suffice, as we want to represent the BSN as a string after serialization.
We also want to deserialize strings directly into Bsns, while still upholding the guarantee that an instantiated Bsn represents a valid BSN.
Therefore, you have to incorporate Bsn::validate into the implementation of the deserialization visitor.
More information on implementing the traits:
serde::Serialize: https://serde.rs/impl-serialize.htmlserde::Deserialize: https://serde.rs/impl-deserialize.html
If everything works out, all tests should pass.
Exercise 3.1.4: 3D Printer
An imaginary 3D printer uses filament to create all kinds of things. Its states can be represented with the following state diagram:
┌─────────────────┐
│ │
│ │ Reset
│ Idle │◄────────────────────────────┐
┌────────►│ │ │
│ │ │ │
│ │ │ │
│ └────────┬────────┘ │
│ │ │
│ │ │
│ │ Start │
│ │ │
│ ▼ │
│ ┌─────────────────┐ ┌────────┴────────┐
│ │ │ │ │
│ │ │ Out of filament │ │
Product │ │ Printing ├──────────────────► │ Error │
retrieved│ │ │ │ │
│ │ │ │ │
│ │ │ │ │
│ └────────┬────────┘ └─────────────────┘
│ │
│ │ Product ready
│ │
│ ▼
│ ┌─────────────────┐
│ │ │
│ │ │
│ │ Product Ready │
└─────────┤ │
│ │
│ │
└─────────────────┘
The printer boots in Idle state. Once a job is started, the printer enters the Printing state. In printing state, it keeps on printing the product until either it is ready or the printer is out of filament. If the printer is out of filament, the printer goes into Error state, which it can only come out of upon device reset. If the product is ready, the printer goes to Product Ready state, and once the user retrieves the product, the printer goes back to Idle.
The printer can be represented in Rust using the typestate pattern as described during the lecture. This allows you to write a simple 3D printer driver. In exercises/3-crate-engineering/1-crate-engineering/4-3d-printer, a Printer3D struct is instantiated. Add methods corresponding to each of the traits, that simulate the state transitions by printing the state. A method simulating checking if the printer is out of filament is provided.
Of course, to make the printer more realistic, you can add more states and transitions.
Unit 3.2 - Testing and Fuzzing
Exercise 3.2.1: DNS Decoding
In this exercise you will practice with unit tests, code coverage, and writing a tiny fuzzer using cargo-fuzz.
You will work with a routine dns_decode that implements a part of RFC1035 DNS name decoding: in this format, all parts of a domain name are encoded by prefixing them with one byte
indicating the length, followed by the bytes that comprise the part. The domain name itself is zero-terminated (i.e., an "empty part" signifies the end of a domain name).
The periods between parts are not encoded.
So, for example, "mailcrab.tweedegolf.nl" can be encoded as b"\x08mailcrab\x0Atweedegolf\x02nl\0". This is subject to two restrictions:
- The maximum size of a part is 63 bytes.
- The maximum size of a full domain name (including periods) is 255 bytes.
We have presupplied a "first attempt" at exercises/3-crate-engineering/2-testing/1-dns-decode. You will find the function definition in src/lib.rs and
a src/main.rs so you can even try the function out interactively. The file src/lib.rs even contains some unit tests, so we are off to a great start! Although this function is
not terribly well-tested and contains some bugs. Maybe you can find them?
Exercise 3.2.1.A Improving Code Coverage
If you haven't done so already, install cargo-llvm-cov, by running:
cargo install cargo-llvm-cov
And then the coverage can be viewed by running cargo llvm-cov from the crate directory. You can also run cargo llvm-cov --open to inspect the coverage report in your browser.
Even though the line coverage is high, as you can see it is not 100%. Write a unit test so that the coverage for src/lib.rs' does reach 100% line coverage (you will obviously rarely hit this in practice).
Exercise 3.2.1.B Fuzz Testing
Even with 100% coverage, there are still bugs lurking in here. Let's find them with cargo-fuzz. To install that, we not only need to install cargo-fuzz, but also use the "nightly" toolchain of Rust, which contains all the experimental features. To install both, run:
rustup toolchain install nightly
cargo install cargo-fuzz
Then, from exercises/3-crate-engineering/2-testing/1-dns-decode, run cargo fuzz init. This will create a subdirectory fuzz/ which contains the template for a fuzzing target.
(Note that this fuzz/ subdirectory is actually a small Rust project, with its own Cargo.toml file! So you can add dependencies to it as well, as with any other Rust project!)
You can list all the available targets using:
cargo +nightly fuzz list
You can already run the target fuzz_target_1 by running
cargo +nightly fuzz run fuzz_target_1
But that will not do find any bugs; edit fuzz/fuzz_targets/fuzz_target_1.rs so it does something interesting! Also consider adding more targets using cargo +nightly fuzz add <TARGETNAME.
Hints:
-
To access the
dns_decodefunction, you need to import it in the fuzzing target using:#![allow(unused)] fn main() { use dns_parse::decode_dns_name; } -
Start simple by writing a fuzzer that just runs
dns_decodeon a&[u8]input and see if it detects crashes. -
Try checking the two other properties that must hold for
dns_decode, that is:- the maximum length of a part is 63 bytes
- the maximum length of a domain name is 255 bytes.
-
As in unit tests,
assert!is your friend.
Unit 3.3 - The Tooling of Cargo
Exercise 3.3.1: FizzBuzz
In this exercise, you will practise writing a unit test, and use Rusts benchmarking functionality to help you optimize a FizzBuzz app. You will need cargo-criterion, a tool that runs benchmarks and creates nice reports. You can install it by running
cargo install cargo-criterion --version=1.1.0
3.3.1.A Testing Fizz Buzz ⭐
Open exercises/3-crate-engineering/3-cargo-tooling/1-fizzbuzz/src/lib.rs. Create a unit test that verifies the correctness of the fizz_buzz function. You can use the include_str macro to include exercises/3-crate-engineering/3-cargo-tooling/1-fizzbuzz/fizzbuzz.out as a &str into your binary. Each line of fizzbuzz.out contains the expected output of the fizz_buzz function given the line number as input. You can run the test with
cargo test
By default, Rusts test harness captures all output and discards it, If you like to debug your test code using print statements, you can run
cargo test -- --nocapture
to prevent the harness from capturing output.
3.3.1.B Benchmarking Fizz Buzz ⭐⭐
You'll probably have noticed the fizz_buzz implementation is not very optimized. We will use criterion to help us benchmark fizz_buzz. To run a benchmark, run the following command when in the exercises/3-crate-engineering/3-cargo-tooling/1-fizzbuzz/ directory:
cargo criterion
This command will run the benchmarks, and report some statistics to your terminal. It also generates HTML reports including graphs that you can find under target/criterion/reports. For instance, target/criterion/reports/index.html is a summary of all benchmark. Open it with your browser and have a look.
Your job is to do some optimization of the fizz_buzz function, and use cargo-criterion to measure the impact of your changes. Don't be afraid to change the signature of fizz_buzz, if, for instance, you want to minimize the number of allocations done by this function. However, make sure that the function is able to correctly produce the output. How fast can you FizzBuzz?
Unit 4.1 - Introduction to Multitasking
There are no exercises for this unit
Unit 4.2 - Parallel Multitasking
Exercise 4.2.1: TF-IDF
Follow the instructions in the comments of exercises/4-multitasking/2-parallel-multitasking/1-tf-idf/src/main.rs!
Exercise 4.2.2: Mutex
The basic mutex performs a spin-loop while waiting to take the lock. That is terribly inefficient. Luckily, your operating system is able to wait until the lock becomes available, and will just put the thread to sleep in the meantime.
This functionality is exposed in the atomic_wait crate. The section on implementing a mutex from "Rust Atomics and Locks" explains how to use it.
- change the
AtomicBoolfor aAtomicU32 - implement
lock. Be careful about spurious wakes: afterwaitreturns, you must stil check the condition - implement unlocking (
Drop for MutexGuard<T>usingwake_one.
The linked chapter goes on to further optimize the mutex. This is technically out of scope for this course, but we won't stop you if you try (and will still try to help if you get stuck)!
Unit 4.3 - Asynchronous Multitasking
Exercise 4.3.1: From sync to async
Synchronous and asynchronous Rust code does not look too different from each other. In this excercise we will turn a synchronous Rust TCP echo server into an asynchronous one.
Open exercises/4-multitasking/3-asynchronous-multitasking/1-sync-to-async in your editor. Follow the steps in main.rs to first test the program works, then convert it to async, and then test it still works.
Exercise 4.3.2: Measurement Data Sink
In this scenario we have a set of IoT sensors that measure air quality in different rooms. They send the data via a TCP socket to a server. The server aggregates the data per room and writes the data to CSV file. The functionality is currently implemented in a synchronous way. Your task is to make the server code async.
Open exercises/4-multitasking/3-asynchronous-multitasking/2-measurement-data-sink in your editor.
In two different terminals run:
cargo run --bin server
and
cargo run --bin sensor-nodes
You should see regular log messages about received measurements. Every 60 seconds new lines should be appended to database.csv.
Exercise 4.3.2A: asyncify
Then address the TODO: comments in src/bin/server.rs. Check that running the application still works as before.
Exercise 4.3.2B: Requirements change
Run the clients with an interval of 10 seconds like this:
cargo run --bin sensor-nodes -- -i 10s
Investigate and address upcoming bugs.
Exercise 4.3.3: Async Channels
Channels are a very useful way to communicate between threads and async tasks. They allow for decoupling your application into many tasks. You'll see how that can come in nicely in exercise E.2. In this exercise, you'll implement two variants: a oneshot channel and a multi-producer-single-consumer (MPSC) channel. If you're up for a challenge, you can write a broadcast channel as well.
4.3.3.A MPSC channel ⭐⭐
A multi-producer-single-consumer (MPSC) channel is a channel that allows for multiple Senders to send many messages to a single Receiver.
Open exercises/4-multitasking/3-asynchronous-multitasking/3-async-channels in your editor. You'll find the scaffolding code there. For part A, you'll work in src/mpsc.rs. Fix the todo!s in that file in order to make the test pass. To test, run:
cargo test -- mpsc
If your tests are stuck, probably either your implementation does not use the Waker correctly, or it returns Poll::Pending where it shouldn't.
4.3.3.B Oneshot channel ⭐⭐⭐
A oneshot is a channel that allows for one Sender to send exactly one message to a single Receiver.
For part B, you'll work in src/broadcast.rs. This time, you'll have to do more yourself. Intended behavior:
ReceiverimplementsFuture. It returnsPoll::Ready(Ok(T))ifinner.dataisSome(T),Poll::Pendingifinner.dataisNone, andPoll::Ready(Err(Error::SenderDropped))if theSenderwas dropped.Receiver::pollreplacesinner.wakerwith the one from theContext.Senderconsumesselfon send, allowing the it to be used no more than once. Sending setsinner.datatoSome(T). It returnsErr(Error::ReceiverDropped(T))if theReceiverwas dropped before sending.Sender::sendwakesinner.wakerafter putting the data ininner.data- Once the
Senderis dropped, it marks itself dropped withinner - Once the
Receiveris dropped, it marks itself dropped withinner - Upon succesfully sending the message, the consumed
Senderis not marked as dropped. Insteadstd::mem::forgetis used to avoid running the destructor.
To test, run:
cargo test -- broadcast
4.3.3.C Broadcast channel (bonus) ⭐⭐⭐⭐
A Broadcast channel is a channel that supports multiple senders and receivers. Each message that is sent by any of the senders, is received by every receiver. Therefore, the implemenentation has to hold on to messages until they have been sent to every receiver that has not yet been dropped. This furthermore implies that the message shoud be cloned upon broadcasting.
For this bonus exercise, we provide no scaffolding. Take your inspiration from the mpsc and oneshot modules, and implement a broadcast module yourself.
Unit 5.1 - Rust for Web Servers
Exercise 5.1.1: Lettuce Crop
In this exercise, we will build a simple web server with axum which allows users to upload images to crop them. You will learn how to serve static HTML pages along with their associated style sheets and images, and you will learn how to handle POST requests with multipart form data to receive the uploaded images.
5.1.1.A Hello axum
In exercises/5-rust-for-web/1-rust-for-web/1-lettuce-crop we have set up the start of our web server. It currently only serves "Hello, world!" for GET requests on the main page. Run the program and go to http://[::]:7000/ in your browser to see if it works.
Note that http://[::]:7000/ is an unspecified wildcard address for IPv6. If you want to use IPv4, you can use http://0.0.0.0:7000/ instead. If you only want to host it on localhost, use http://127.0.0.1:7000/ or http://[::1]:7000/ instead.
In main.rs you can see the Router that is used to serve "Hello, world!". We can chain multiple routes to serve multiple end-points. Try adding a second route which serves GET requests on another page (e.g. /hello).
5.1.1.B Serving static files
Currently, our web server only serves static strings. To serve static HTML documents, CSS style sheets, images and other files, we will use the ServeDir file server from tower_http. We can add this file server to our router as a fallback service to resolve any request which does not match any other defined route with our file server.
Add a fallback_service to the router with a ServeDir that serves files from the assets folder.
If you now go to http://0.0.0.0:7000/index.html you should see the Lettuce Crop web page with appropriate styling and an image of a lettuce.
By default, ServeDir will automatically append index.html when requesting a path that leads to a directory. This means that if you remove the "Hello, world!" route for / from the router, you will also see the Lettuce Crop page on the main page of the website!
5.1.1.C POST requests and dynamic responses
On the Lettuce Crop page we have set up an HTML form, which when submitted sends a POST request to /crop:
<form action="/crop" method="post" enctype="multipart/form-data">
POST requests are requests that can contain additional data to send to the server. In this case, the form data, consisting of an image and the max size value, will be sent along with the request.
If you select an image and press the crop button, you will be redirected to /crop, which currently does not exist. If you open the browser's developer tools (right click > Inspect, or ctrl+shift+i) and go to the network tab, you should see the POST request which currently returns status 405 (Method Not Allowed). The /crop route is currently handled by our fallback service, which does not accept POST requests. If you go to http://0.0.0.0:7000/crop directly without using the form, the browser will instead send a regular GET request, which will return status 404 (Not Found).
Let's add a route for /crop to our router which will handle the POST requests from the form. You can specify the route in the same way as we did for GET requests, but using post instead of get.
Instead of returning a static string, we can also use a function to respond to requests. Define the following function and pass it to the post method for /crop:
#![allow(unused)] fn main() { async fn crop_image() -> String { format!("Hi! This is still a work in progress. {}", 42) } }
5.1.1.D Handling uploaded files (multipart form data)
So how do we get the form data from the POST request? With axum, we use extractors to get information about the request, such as headers, path names or query parameters. Normally, we would use the Form extractor to get the submitted form data. However, because we want the user to be able to upload an image, we use the multipart form data encoding, as specified by the enctype in the HTML form tag.
To extract multipart form data in axum, we use the Multipart extractor. Unlike the Form extractor, the Multipart extractor does not automatically deserialize the data into a convenient struct. Instead, we will have to manually loop through the fields and deserialize the data we need.
Add mut multipart: Multipart as a parameter to our crop_image function to extract the multipart form data. Then, use the following loop to print all available fields that were included in the POST request:
#![allow(unused)] fn main() { while let Some(field) = multipart.next_field().await.unwrap() { let name = field.name().unwrap().to_string(); let bytes = field.bytes().await.unwrap(); println!("{name}: {} bytes long", bytes.len()); } }
Once you submit the form, it should show an image field containing the image data and a max_size field corresponding to the max size number input field in the form.
Let's deserialize the two form fields:
-
The
imagefield consists of the bytes that make up the image. We will use anImageReaderfrom theimagecrate to read the image data:#![allow(unused)] fn main() { ImageReader::new(Cursor::new(bytes)).with_guessed_format().unwrap().decode() }This will return a
DynamicImage, which can be a variety of different image formats. With theimagecrate we will be able to crop and resize this image. -
The
max_sizefield contains a number encoded a plain text. You can retrieve the text usingfield.text()instead offield.bytes(), and you can parse it into a number using.parse(). Let's make it au32.
We will leave it up to you to implement the logic to deserialize these two fields and turn them into a DynamicImage and a u32 that can be used after we're done looping through all the fields.
Change the string returned by crop_image to the following to verify that it works:
#![allow(unused)] fn main() { format!("Image size: {}x{}\nMax size: {}", image.width(), image.height(), max_size) }
5.1.1.E Sending the cropped image as response
Let's crop the DynamicImage into a square using the following code:
#![allow(unused)] fn main() { let size = min(min(image.width(), image.height()), max_size); let image = image.resize_to_fill(size, size, imageops::FilterType::Triangle); }
The size of the cropped square image is the minimum of the image's width, height and the configured maximum size. The resize_to_fill method will crop and resize the image to our size and center it appropriately.
Now that we have cropped the image, we need to send it back to the client. We encode the image back into an image format with write_to; we've chosen to return the cropped images as WebP's:
#![allow(unused)] fn main() { let mut image_buffer = Vec::new(); image .write_to(&mut BufWriter::new(Cursor::new(&mut image_buffer)), ImageFormat::WebP) .unwrap(); }
To send these bytes as an image to the client, we will have to create a response with a proper content type header and our image buffer as a body. Update the crop_image to return a Response instead of a String, and construct a response with Response::builder(). Set the "content-type" header to match your chosen image format (for example image/webp for WebP images), and construct a body from the image buffer using Body::from.
If you now submit an image on the site, it should be returned to you cropped into a square!
5.1.1.F Error handling & input validation
Currently, the handler likely contains many .unwrap()s, which may panic. Luckily, axum catches these panics from our handler and will keep running after printing the panic. However, the user will not get any proper response from axum when these panics happen. To give the client some feedback about what went wrong, we can implement some better error handling.
Let's change our crop_image function to return a Result<Response, (StatusCode, &'static str)>. This gives us the ability to return errors consisting of an HTTP status code and a static string.
For example, let's say the user uploads a corrupted image. Then, the .decode() method of our ImageReader will return an error, causing the .unwrap() to panic. Let's replace the .unwrap() with a .map_err that notifies the user that they did a bad request:
.map_err(|_| (StatusCode::BAD_REQUEST, "Error: Could not decode image"))?
Similarly, you can also add appropriate error handling in other places, returning appropriate HTTP status codes.
Currently, the size of our cropped image is defined as the minimum of the original image's width and height, and the set max_size value. The max_size value has a maximum of 2048 set in the HTML form. However, you should never trust the data coming from the client-side as HTML and JavaScript code running on the client's device can easily be modified, and the client can send modified HTTP requests. So let's also return a StatusCode::BAD_REQUEST if max_size is larger than 2048.
By default, there is a 2 MB limit for request bodies. If a user submits an image larger than this limit, the .bytes() call on the multipart field will return an error. In this case, we could return a StatusCode::PAYLOAD_TOO_LARGE. If you want to accept larger images, you can configure a larger limit by setting a custom DefaultBodyLimit.
5.1.1.G Serving files from memory (bonus)
Currently, the static files are served from the assets folder. Instead, we can also bundle these files into the binary with memory-serve. Not only is it convenient to bundle all files into a single binary, but it can also improve performance!
After adding memory-serve to your project with cargo add memory-serve, we can define a memory router as follows:
#![allow(unused)] fn main() { let memory_router = MemoryServe::new(load_assets!("assets")) .index_file(Some("/index.html")) .into_router(); }
Now we can use this memory router as fallback service instead of the ServeDir.
If you build the project in release mode (cargo build --release), you will see the files in the assets folder being included in the binary!
Exercise 5.1.2: Pastebin
This exercise is about writing a simple pastebin web server. The web server will again be powered by axum. For this exercise, you will need to set up the project yourself.
- Data is kept in memory. Bonus if you use a database or
sqlite, but first make the app function properly without. - Expose a route to which a POST request can be sent, that accepts some plain text, and stores it along with a freshly generated UUID. The UUID is sent in the response. You can use the
uuidcrate to generate UUIDs. - Expose a route to which a GET request can be sent, that accepts a UUID and returns the plain text corresponding to the UUID, or a 404 error if it doesn't exist.
- Expose a route to which a DELETE request can be sent, that accepts a UUID and deletes the plain text corresponding to that UUID.
Unit 5.2 - Rust in the Cloud
In this section, we will show how you can use Rust to create cloud services. The first exercise uses Amazon's AWS, and the second exercise uses European cloud service provider Scaleway.
Exercise 5.2.1: Lettuce Crop AWS
In this exercise, we will port our Lettuce Crop website from exercise 5.1.1 to the cloud using AWS Lambda. AWS Lambda allows you to run code in the cloud in a serverless configuration. This means that machines in Amazon's data centers will automatically start running your code when needed, which means you do not have to worry about managing servers, and you only pay for the compute time you use.
For this exercise, the AWS free tier should be sufficient. However, please do remember to shut off your Lambdas once you are done testing to avoid any unexpected costs! See the free tier page in the billing and cost management section of the AWS console to see how much of the free tier quotas you have left this month.
5.2.1.A Setting up Cargo Lambda
To build for AWS Lambda with Rust, we will use Cargo Lambda. You can install Cargo Lambda with Cargo Binstall:
cargo binstall cargo-lambda
You may also need to install Zig, which is used for cross-compilation. Cargo Lambda will inform you if Zig is not installed when building your Lambda, in which case it will attempt to help you install it automatically via pip or npm.
Alternatively, you can use any of the other installation methods for Cargo Lambda found here.
5.2.1.B Axum router with Lambda HTTP
The lambda_runtime crate provides the runtime for AWS Lambdas written in Rust. The lambda_http crate provides an abstraction layer on top of the lambda_runtime to make it easy to develop HTTP servers on AWS Lambda with Rust, which is ideal for small dynamic websites or REST APIs.
Add lambda_http to the Rust project with:
cargo add lambda_http
Since lambda_http is able to run axum routers, we only really need to change the main function to convert our Lettuce Crop server to a Lambda. We create our Router as usual, but instead of serving it with axum::serve, we run the Router with the run function from lambda_http:
use lambda_http::{run, tracing, Error}; #[tokio::main] async fn main() -> Result<(), Error> { // required to enable CloudWatch error logging by the runtime tracing::init_default_subscriber(); let app = Router::new() .route("/crop", post(crop_image)) .fallback_service(ServeDir::new("assets")); run(app).await }
Update the main function as above and then try out the Lambda with:
cargo lambda watch
This will emulate the Lambda locally on your device, serving it on http://localhost:9000/ by default.
5.2.1.C Setting up a Lambda function in the AWS console
Now that we've tested our Lambda locally, let's create a Lambda function in the AWS console. Go to the AWS Lambda page in the AWS console, and click "Create a function". Then, configure it as follows:
- Select "Author from scratch"
- Give it a name, for example "lettuce-crop"
- Select the "Amazon Linux 2023" runtime
- Select "arm64" architecture (which offers lower costs compared to x86_64)
- In "Additional Configurations" enable "Enable function URL", and select Auth type "NONE" to get a publicly accessible URL for your Lambda function
Finally, click "Create function" and wait a few seconds for your Lambda to be created.
5.2.1.D Building & deploying our Lambda function
Before we deploy our Lambda, we first have to build our project with the appropriate architecture:
cargo lambda build --release --arm64 --output-format zip
This will generate a bootstrap.zip in the target/lambda/{project name} folder, which we can upload in the AWS console to deploy our Lambda.
However, this zip file does not contain our assets. If we want our Lambda to be able to serve our HTML document and the corresponding CSS file and image, we have to include these assets. Let's create a CargoLambda.toml config file to specify how we want to build our Lambda, and include the following:
[build]
arm64 = true
output_format = "zip"
include = ["assets/index.html", "assets/styles.css", "assets/crop.webp"]
If we now build our Lambda with cargo lambda build --release we will get a zip that also contains our assets (we no longer need the --arm64 and --output-format command line arguments, as these are now set in our config file).
Alternatively, if you are using memory-serve to serve the assets, as described in exercise 5.1.1.G, you will not need to include the assets in the zip, as they already will be included in the binary.
To deploy the Lambda, click the "Upload from" button in the "Code" tab for our Lambda in the AWS console. Then, upload the bootstrap.zip file. Now, the Lambda should be live! Open the function URL listed in the function overview at the top of the page to try it out!
You can also use cargo lambda deploy to deploy your Lambda via the CLI. However, this does require you to set up AWS credentials first.
Note that AWS Lambda only accepts files up to 50 MB, for larger projects you can instead upload to an S3 bucket. S3 does not have a free tier, but it does have a 12-month free trial.
5.2.1.E Analyzing Lambda usage via CloudWatch
Now that our Lambda is up and running, let's take a look around the AWS console. If you go to the "Monitor" tab, you can see some metrics about the requests handled by the Lambda function. These basic metrics are automatically gathered by CloudWatch free of charge.
If you scroll down to CloudWatch Logs, you will see recent invocations of the Lambda function. If you click on the log stream of one of these requests, you will see the logs produced while handling the request. The outputs from any println!'s or logs from the tracing crate should show up here. The free tier of CloudWatch allows you to store up to 5 GB of log data for free.
You can also see a list of the most expensive invocations on the "Monitor" tab. The cost is measured in gigabyte-seconds, which is the amount of memory used for the duration it took to handle the request. The free tier for AWS Lambda gives you 1,000,000 requests and 400,000 gigabyte-seconds for free per month.
By default, Lambdas are configured with 128 MB of memory, which can be increased in the "Configuration" tab (but it cannot be set lower than 128 MB). In this tab you can also configure the timeout for handling requests. By default, the Lambda will time out after 3 seconds, but this can be changed if needed.
Where to go from here?
- The Rust Runtime for AWS Lambda GitHub repository contains a bunch of useful examples, which show for example how to interact with S3 buckets or how to create Lambda extensions.
- The AWS SDK for Rust allows you to interact with AWS services via Rust.
Remember to throttle or delete the Lambda function once you are done testing to prevent unexpected costs!
Exercise 5.2.2: Lettuce Crop Scaleway
In this exercise, we will port our Lettuce Crop website from exercise 5.1.1 to the cloud using Scaleway Serverless Functions. Scaleway Serverless Functions allows you to run code in the cloud in a serverless configuration. This means that machines in Scaleway's data centers will automatically start running your code when needed, which means you do not have to worry about managing servers, and you only pay for the compute time you use.
For this exercise, the free resources offered by Scaleway should be sufficient. However, please do remember to shut off your serverless functions once you are done testing to avoid any unexpected costs! See the serverless pricing page for up-to-date information about the costs associated with running serverless functions.
5.2.2.A General project structure
Serverless functions on Scaleway work by providing a handler function that takes a request and returns a response. In exercises/5-rust-for-web/2-rust-in-the-cloud/2-lettuce-crop-scaleway we have laid out the basic project structure for a Scaleway serverless function. In src/handler.rs there is a handler function that turns any request into a response with the body "Hello, world!".
To expose this function to Scaleway, we must configure our project as a library, using src/handler.rs as the main entry point. To do this, add the following to the Cargo.toml:
[lib]
path = "src/handler.rs"
5.2.2.B Deploying the serverless function
To deploy our serverless function, go to the functions page in the Scaleway console, select your preferred region, and pick a namespace (or create a new one). In this namespace, click the "Create function" button and select Rust as the runtime.
To upload our code, we first create a zip file with the src directory and the Cargo.toml. Upload this zip to Scaleway, and set the handler to the name of the function in src/handler.rs that you want to use to handle requests (my_handler in our case).
Now you can set a name and configure to resources as needed, and then you can create the function. Wait for it to deploy (which might take a couple of minutes), and then go to the function endpoint URL to see the serverless function in action.
5.2.2.C Request handling with axum routing
To add some more interesting functionality to this handler, we can set up an axum router like we did in exercise 5.1.1. However, with serverless functions, we cannot serve the router using axum::serve. Instead, we can call the router as a service with the currently incoming request:
let mut router = Router::new()
.route("/", get("Hello, world!"))
.route("/crop", get("Lettuce crop!"));
router.as_service().call(req).await.unwrap()
This call method will then provide a response based on the request using the specified routes.
Go ahead and modify the handler to use a router with a couple of different routes, and redeploy it on Scaleway to test it.
5.2.2.D Serving files from serverless functions
To be able to host Lettuce Crop on a Scaleway serverless function, we need to be able to serve the static assets for the website. Normally, we could use ServeDir for this, but Scaleway's serverless function don't have access to files during runtime. Instead, we can include the files in the binary using memory-serve, as described in exercise 5.1.1.G.
Copy over the assets folder and the crop_image function from exercise 5.1.1, and configure the router to crop images sent with POST requests to /crop, and serve the assets using memory-serve as a fallback service. If you now redeploy your code (don't forget to also include the assets folder in the zip), we should now have a fully working version of Lettuce Crop on Scaleway!
Note that in real world applications, it is better (and cheaper!) to host the static parts of the webpage separately using e.g. storage buckets, and only handle the interactive part (i.e. the post requests to crop images) in the serverless function.
Unit 5.3 - Rust in the Browser
Exercise 5.3.1: Lettuce Crop WebAssembly
In exercise 5.1.1, we build a web server that hosts an image cropping service. But do we really need to do this cropping on our server? Wouldn't it be much more privacy-friendly if we could do the image cropping in the user's browser instead of uploading images to our external server?
In this exercise, we will create a new version of our Lettuce Crop website that crops images with WebAssembly. WebAssembly allows you to run compiled code in a safe sandboxed environment in the browser. This means we will not need a dedicated server anymore, as the website will only consist of static files which we can be hosted using any HTTP server. You could even host it for free using GitHub pages!
5.3.1.A Building with Wasm Pack
In exercises/5-rust-for-web/3-rust-in-the-browser/1-lettuce-crop-wasm we have set up a basic WebAssembly project. As you can see in the Cargo.toml, the project has been configured as a dynamic library ("cdylib"). We've also added the wasm-bindgen crate as a dependency, which is used to generate WebAssembly bindings.
To build the project, we will use wasm-pack. First, install wasm-pack with:
cargo install wasm-pack
Then, build the project with wasm-pack. Since we want to use it in the browser, we set the wasm-pack target to web, and we tell it to put the generate files in the assets/pkg folder:
wasm-pack build --target web --out-dir assets/pkg
Now, a bunch of files should appear in the assets/pkg folder:
- A
.wasmfile, which contains the compiled WebAssembly code - Some
.d.tsfiles, which describe the TypeScript types of the generated bindings - A
.jsfile, which contains the JavaScript bindings for our WebAssembly binary
5.3.1.B Interacting with JavaScript
So what functionality does the compiled WebAssembly currently include? In lib.rs you can see two functions: an extern alert() function, and a hello() function. Both of these functions have been annotated with #[wasm_bindgen] to indicate that we want to bind them with WebAssembly. Extern functions will be bound to existing JavaScript methods, in this case the window's alert() function which shows a popup dialog.
Let's add the WebAssembly to our website. Add the following JavaScript in the <body> of the index.html to load the WebAssembly binary and call our hello() function when we press the submit button:
<script type="module">
import init, { hello } from "./pkg/lettuce_crop_wasm.js";
init().then(() => {
const submit_button = document.querySelector('input[type="submit"]');
submit_button.onclick = () => {
hello("WebAssembly");
}
});
</script>
To try out the website, you can use any HTTP server that is able to serve local files. You could use axum to host the files like we did in exercise 5.1.1, but you can also use for example npx http-server if you have npm installed.
5.3.1.C Cropping images
Let's add a crop_image(bytes: Vec<u8>, max_size: u32) -> Vec<u8> function to our Rust library that will crop our images. You can use the same logic as in exercise 5.1.1 (part D and E) to create a DynamicImage from the input bytes, crop it, and export it as WebP. Mark the function with #[wasm_bindgen] and rebuild the library to generate WebAssembly bindings for it.
If you look at the generated JavaScript bindings, you will see that the Vec<u8>s for the crop_image function have been turned into Uint8Arrays. We will need to write some JavaScript to read the user's selected image and give it to our crop_image as a Uint8Array.
First, let's grab our other two input elements:
const max_size = document.querySelector('input[name="max_size"]');
const image = document.querySelector('input[name="image"]');
Then, in the onclick of the submit button, you can grab the selected file using image.files[0]. To get the contents of the file, we will use a FileReader:
const file = image.files[0];
const reader = new FileReader();
reader.onload = (evt) => {
const bytes = new Uint8Array(evt.target.result);
const cropped_bytes = crop_image(bytes, max_size.value); // call our function
// TODO: do something with the cropped_bytes
};
reader.readAsArrayBuffer(file);
Finally, to display the resulting cropped image to the user, we will construct a Blob from the Uint8Array, and turn this Blob into a URL to which we will redirect the user:
window.location.href = URL.createObjectURL(new Blob([cropped_bytes]));
If you select an invalid file, you will get an error in the browser console. Feel free to add some better error handling by using a try-catch, and by validating whether image.files[0] exists before reading it. It would also be nice to verify that max_size has a sensible value.
5.3.1.D Using the web-sys crate (bonus)
Instead of using JavaScript to interact with the HTML document or manually binding extern JavaScript functions using #[wasm_bindgen] like we saw with alert(), we can also use the web-sys crate. This crate provides bindings for the JavaScript web APIs available in the browser. However, most of these APIs have to be manually enabled with individual features.
Add the web-sys crate to your project with all the needed features enabled:
cargo add web-sys --features "Window,Document,HtmlElement,HtmlImageElement,Blob,Url"
Now, instead having the crop_image function return an array of bytes, let's have it instead append an image to HTML document:
- First, get the HTML body element:
#![allow(unused)] fn main() { let window = web_sys::window().unwrap(); let document = window.document().unwrap(); let body = document.body().unwrap(); } - Then, we can create an HTML image element:
#![allow(unused)] fn main() { let img = document.create_element("img").unwrap(); let img: web_sys::HtmlImageElement = img.dyn_into().unwrap(); } - To set the source of the image, we will again need to create a
Blobto get a temporary data URL. For this, we first create a JavaScript array:#![allow(unused)] fn main() { let bytes = web_sys::js_sys::Array::new(); bytes.push(&web_sys::js_sys::Uint8Array::from(&buffer[..])); } - And then we can create a Blob and create a URL:
#![allow(unused)] fn main() { let blob = web_sys::Blob::new_with_u8_array_sequence(&bytes).unwrap(); let url = web_sys::Url::create_object_url_with_blob(&blob).unwrap(); } - And finally, we can set the image's source and append the image to the document's body:
#![allow(unused)] fn main() { img.set_src(&url); body.append_child(&img).unwrap(); } - Remember to also update the JavaScript code in the HTML document accordingly.
Unit 6.1 - Foreign Function Interface
Exercise 6.1.1: CRC in C
In this exercise, we will call a CRC checksum function written in C from a Rust program.
Instructions
Prerequisites:
- A C compiler
Steps:
-
Add the
ccbuild dependency, by adding toCargo.tomlthe lines:[build-dependencies] cc = "1.0" -
Create
build.rs(in the same directory asCargo.toml) with contentsfn main() { println!("cargo:rerun-if-changed=crc32.h"); println!("cargo:rerun-if-changed=crc32.c"); cc::Build::new().file("crc32.c").compile("crc32"); }build.rsis a build script that cargo runs before it compiles your crate. This will find your c code, compile it, and link it into the executable rust produces. -
In
main.rs, define an extern (fill in the argument and return types)#![allow(unused)] fn main() { extern "C" { fn CRC32( ... ) -> ...; // hint: https://doc.rust-lang.org/std/ffi/ } } -
Now, create a rust wrapper that calls the extern function
#![allow(unused)] fn main() { fn crc32( ... ) -> ... { ... // (hints: `unsafe`, `.as_ptr()`, `.len()`) } } -
Call our wrapper on some example input
fn main() { println!("{:#x}", crc32(b"abcde")); }In the above example, the correct output is
0x8587d865
Exercise 6.1.2: CRC in Rust
Now, let's do it the other way around: we can use a CRC checksum function written in Rust in a C program.
Instructions
Prerequisites:
- A C compiler
Steps:
-
Add this to Cargo.toml
[lib] name = "crc_in_rust" crate-type = ["staticlib"] -
Expose an extern rust function in the
lib.rs#![allow(unused)] fn main() { #[unsafe(no_mangle)] pub extern "C" fn crc32(...) -> ... { ... crc32_rust(...) } } -
Create a C header file
crc_in_rust.h#include <stdint.h> // uint32_t, uint8_t #include <stddef.h> // size_t uint32_t crc32(const uint8_t* data, size_t data_length); -
Create
main.cand use the rustcrc32function#include "crc_in_rust.h" #include <stddef.h> // size_t #include <inttypes.h> // uint32_t, uint8_t, PRIx32 #include <stdio.h> // printf int main() { uint8_t data[] = "abcde"; size_t data_length = 5; uint32_t hash = crc32(data, data_length); printf("Hash: 0x%"PRIx32"\n", hash); return 0; } -
Give the rust function the same signature as the one defined in the header file
-
Compile the rust crate and then run
Linux & MacOS:
# Build main.c, link it to the dynamic library and output the executable called main $ clang main.c target/debug/libcrc_in_rust.a -lpthread -ldl -omain # Run the executable $ ./main Hash: 0x8587d865Windows:
# Build main.c, link it to the import library of the DLL and output the executable called main.exe ❯ clang main.c .\target\debug\crc_in_rust.lib -lkernel32 -lntdll -luserenv -lws2_32 -ldbghelp -o "main.exe" # Run the executable ❯ .\main.exe Hash: 0x8587d865
Exercise 6.1.3: QOI Bindgen
In this exercise, we will use cargo bindgen to generate the FFI bindings for a C library. Bindgen will look at a C header file, and generate Rust functions, types and constants based on the C definitions.
However, the generated code will likely be ugly and non-idiomatic. To wrap a C library properly, good API design and documentation is needed.
Background
The image crate provides functionality for encoding, decoding and editing images in Rust. It supports many image formats, like JPEG, PNG and GIF, but also QOI. QOI is a "Quite OK Image format", which aims for fast encoding and decoding of images, while providing a file size similar to PNGs. In this exercise, we test if the image crate produces the same results when decoding QOI images as the QOI reference C library.
The QOI C library is a header-only library, which means the function implementations are included within the header file instead of in a separate C file. We've added a separate C file which includes the header to make it easier to compile and include the library in our Rust program.
Generating bindings
Prerequisites:
- A C compiler is installed on the system
- Bindgen, which can be installed with
cargo install bindgen-cli
Steps:
-
Create the Rust bindings:
bindgen qoi.h -o src/bindings.rs -
Use a
build.rsscript to compile and linkqoi.h. Createbuild.rsand insertfn main() { cc::Build::new().file("qoi.c").compile("qoi"); // outputs `qoi.a` }And add this section to your
Cargo.toml[build-dependencies] cc = "1" -
Create
src/lib.rswith the contentspub mod bindings;. This will make thebindingsmodule available inmain.rs. -
Run
cargo checkto verify everything is compiling correctly.
Inspecting our bindings
In the generated bindings.rs file we find this signature for the qoi_read C function from QOI:
#![allow(unused)] fn main() { extern "C" { pub fn qoi_read( filename: *const ::std::os::raw::c_char, desc: *mut qoi_desc, channels: ::std::os::raw::c_int, ) -> *mut ::std::os::raw::c_void; } }
Some observations:
- The definition is inside an
extern "C"block, and has no body. Therefore, this function is marked as an extern, and Rust expects it to be linked in. - The function is marked
pub, meaning we can import and use it in other modules (likemain.rsin our case) - We can deduce the behavior somewhat from the type signature:
filenameis a C string with the name of the QOI file we want to readdescdescribes some metadata about the image, the function will write to thisqoi_descstruct. This struct was also generated by bindgen:#![allow(unused)] fn main() { #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct qoi_desc { pub width: ::std::os::raw::c_uint, pub height: ::std::os::raw::c_uint, pub channels: ::std::os::raw::c_uchar, pub colorspace: ::std::os::raw::c_uchar, } }channelsis the number of channels the image has: either 3 for RGB images, or 4 for RGBA images (which also have an alpha channel for transparency). For this exercise, we will assume the images have an alpha channel.- The return value is a void pointer. If the function has successfully read the pixel data from a QOI image, then this pointer should point towards the pixel data.
- As the types are raw C types, it can be a hassle to call it directly from Rust.
We will deal with the last point by writing a nice Rust wrapper around the generated bindings.
Writing our wrapper
To make the qoi_read function easier to use, we would like to write a wrapper that takes a path and returns an image buffer:
#![allow(unused)] fn main() { fn read_qoi_image(filename: &Path) -> ImageBuffer<Rgba<u8>, &[u8]> { todo!() } }
To implement this wrapper, there are a couple of challenges that need to be solved:
- We need to turn the path into a C string. Hint: we can use
std::ffi::CString::newto create a C string from a sequence of bytes, and the most convenient way to turn the path into bytes is to first get theOsStrfrom it. We can then pass the C string as a pointer. - We need to provide a
qoi_desc, this struct can be imported from the bindings. Pass a mutable reference to an instance of this struct to the function. - After calling
qoi_read, we need to turn the returned void pointer into an image buffer.- First, we should check if the returned void pointer
is_null(). If it is null, something has gone wrong with reading the image. - Next, we need to determine the length of the returned pixel data. Assuming the image has an alpha channel, we have 4 bytes for every pixel in the image. The number of pixels in the image can be determined from the
qoi_descstruct. - Now we can turn our void pointer into a
&[u8]. We can cast our void pointeras *const u8first. Next, we usestd::slice::from_raw_partswith the previously calculated length. - Finally, we can use
ImageBuffer::from_rawto construct our image buffer.
- First, we should check if the returned void pointer
To try out our wrapper, we can try to read a QOI image and export it as a PNG:
fn main() { let image = read_qoi_image(Path::new("image.qoi")); image.save("image.png").unwrap(); }
If implemented correctly, this should produce a nice picture!
Now that we can decode images using the QOI reference C library, we can test if the image crate produces the same results with the following unit test:
#![allow(unused)] fn main() { #[cfg(test)] mod tests { use crate::read_qoi_image; use std::path::Path; #[test] fn test_qoi_read() { let filename = "image.qoi"; let image = image::open(filename).unwrap().into_rgba8(); let my_image = read_qoi_image(Path::new(filename)); assert_eq!(image.width(), my_image.width()); assert_eq!(image.height(), my_image.height()); assert!(image.pixels().eq(my_image.pixels())); } } }
If you add this test to main.rs and run it with cargo test we should see:
running 1 test
test tests::test_qoi_read ... ok
Freeing the pixel data
When working with data from C, we are responsible for deallocating the memory once we are done using it. Some C libraries might provide a separate function to clean up data structures. For QOI, we instead have to call libc::free to free the memory, as indicated by the documentation of the qoi_read function:
The returned pixel data should be free()d after use.
To make sure someone using our wrapper does not forget to free the memory, we can implement the Drop trait to automatically call libc::free when the variable goes out of scope.
- First, create a wrapper
struct MyImage<'a>(ImageBuffer<Rgba<u8>, &'a [u8]>);, which holds the image buffer. - Next, implement the
Droptrait forMyImageto free the memory (we should retrieve the pointer from the image buffer and cast it back to a void pointer):#![allow(unused)] fn main() { impl Drop for MyImage<'_> { fn drop(&mut self) { todo!(); // call libc::free here using a pointer to the image buffer } } } - To make this
MyImagewrapper more convenient to use, we can also implement theDereftrait to allow us to directly call the methods from the internal image buffer on it:#![allow(unused)] fn main() { impl<'a> Deref for MyImage<'a> { type Target = ImageBuffer<Rgba<u8>, &'a [u8]>; fn deref(&self) -> &Self::Target { &self.0 } } } - Now update the
read_qoi_imagefunction to return an instance ofMyImage.
Uninitialized memory
There is one more trick: our current function initializes the qoi_desc struct with zeros (or whatever values you put there while creating an instance of the struct). This is wasteful because the extern function will overwrite these values. Because the extern function is linked in, the compiler likely does not have enough information to optimize this.
For a relatively small struct such as qoi_desc, this is not much of a problem. However, for larger structures or big arrays, this can make a serious impact on performance.
If we look at the LLVM IR, the intermediate representation which is generated and optimized before it gets turned into assembly code, we can see that it did not optimize away the initialization of the struct with values. Here we see it uses memset to initialize the desc with zeros before calling qoi_read:
call void @llvm.memset.p0.i64(ptr noundef nonnull align 4 dereferenceable(10) %desc.i, i8 0, i64 10, i1 false), !noalias !142
%pointer.i = call noundef ptr @qoi_read(ptr noundef nonnull %t.0.i.i, ptr noundef nonnull %desc.i, i32 noundef 4) #17, !noalias !142
(The LLVM IR can be generated using cargo rustc --bin qoi-bindgen --release -- --emit=llvm-ir)
The solution is to use std::mem::MaybeUninit:
#![allow(unused)] fn main() { let mut desc = MaybeUninit::uninit(); let pointer = unsafe { qoi_read(filename.as_ptr(), desc.as_mut_ptr(), 4) }; let desc = unsafe { desc.assume_init() }; }
The MaybeUninit type is an abstraction for uninitialized memory. The .uninit() method gives a chunk of uninitialized memory big enough to store a value of the desired type (in our case qoi_desc will be inferred).
Conclusion
In this exercise we saw how we can generate bindings to a C library with bindgen. The generated bindings are a bit difficult to work with, as they are unsafe and rely on C types. We've discussed how we can create nice wrappers around the generated bindings to deal with all these C types and to make them safer to work with.
Unit 7.1 - Rust from Python
Exercise 7.1.1: Test your environment
Install uv (https://docs.astral.sh/uv/#installation) and Rust (https://www.rust-lang.org/tools/install)
Check it is working:
cd rust-dna
uv run maturin develop --uv
cd ..
uv run pytest
The output should look something like this:
❯ uv run pytest
Installed 5 packages in 9ms
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.3.5, pluggy-1.5.0
rootdir: /home/tamme/dev/rust-training/exercises/7-rust-for-data-science/1-rust-from-python
configfile: pyproject.toml
collected 10 items
test_decoding.py .... [ 40%]
test_kmers.py .... [ 80%]
test_setup.py . [ 90%]
test_validation.py . [100%]
============================== 10 passed in 0.06s ==============================
Exercise 7.1.2: Translating Python to rust
Follow the steps in exercises/7-rust-for-data-science/1-rust-from-python/main.py to translate the logic to Rust in exercises/7-rust-for-data-science/1-rust-from-python/rust-dna/lib.rs.
Unit 7.2 - Rust from Python
Exercise 7.2.1: Test your environment
7.2.1.A: Set up the tools ⭐
Follow the instructions to instal the Maturin build tool: https://pyo3.rs/v0.21.2/getting-started. We recommend using pyenv: https://github.com/pyenv/pyenv, but pyenv only supports UNIX and WSL. Of course, you can use your favourite venv manager, too.
Navigate to the exercises/7-rust-for-data-science/1-rust-from-python/1-hello-world/ folder in your terminal. Setup and activate your virtual environment, and install the things we need. If you're using pyenv, that means running the following commands:
$ pyenv activate pyo3
$ pip install maturin
$ pip install asyncio
$ pip install aiofiles
7.2.1.B: Trying it out ⭐
If everything went well, you should now be able to build this project into a Python extension:
$ maturin develop
$ python test.py
The test.py script ends in a call to a function that unconditionally raises, so your script is expected to result in an uncaught exception.
Now, try running test_async.py as well and observe its behavior.
7.2.1.C: Running futures concurrently ⭐⭐
You can use asyncio.gather in your python script to execute multiple awaitables. Spawn a number of print_sleep futures, each with different sleep durations using asyncio.gather and observe the behavior.
Exercise 7.2.2: Streaming JSON
Here's a cool crate: Struson. It's a crate that allows for (de)serialization of JSON objects in a streaming fashion, meaning that you don't need to hold all of the data in memory. This contrasts a bit with the great library serde_json, which is very, very ergonomic, but does not allow for easy streaming (de)serialization of data. We're going build something around Struson, that we can use from Python.
Now, let's say we have a stream of JSON. Could be from someone on the internet or some other process running on the machine. Or, you know, a file. The stream consists of data that looks like this:
[
{
"lhs": {
"d":[
1, 2, 3, 4,
5, 6, 7, 8,
9, 10, 11, 12
],
"n": 3
},
"op": [
{
"code": "dot",
"rhs": {
"d": [
13, 14, 15, 16,
17, 18, 19, 20,
21, 22, 23, 24
],
"n": 3
}
}
]
}
]
This JSON is an array of objects that each represent the following:
lhs: a matrix withmrows andncolumns.mcan be derived fromnand the length ofd. In this casem = len / n = 12 / 3 = 4.op: a sequence of operations that need to take place givenlhsand, if the operation takes two operands, itsrhsfield.xcorresponds to the operation that should be executed, and should more or less correspond to the methods provided on nalgebra's Matrix type. In this case, theMatrix::dotmethod should be run, which, for probably good reason, is describead as 'the dot product between two vectors or matrices (seen as vectors)'.
Our library should streamingly deserialize each incoming object from an asyncio stream of bytes, apply the given operation, and pass on the result.
Exercise 7.2.2.A: Give it a go ⭐
The scaffolding code in exercises/7-rust-for-data-science/1-rust-from-python/2-strompy/ implements what we want in Rust, and also provides a synchronous implementation. Have a look around in the code, and try to make sense of it. https://docs.rs is your friend if you want to know more about the dependencies that are being used. Note that for struson, we import a fork that supports deserializing asynchronously.
You can run cargo test to build and run the tests in src/lib.rs.
Exercise 7.2.2.B: The sync way ⭐⭐
During the next exercises, it's good to keep the PyO3 User Guide and the PyO3 API docs at hand.
This exercise is about making strompy_test.py run. Therefore, fix the todo!() in the exec function in src/lib.rs. Don't forget to add the function to the exposed module!
If you've done it correctly, you should be able to run it with the following commands:
$ maturin develop
$ python strompy_test.py
7.2.2.B: The async way ⭐⭐⭐
This time, we'll make strompy_test_async.py work. Implement feed_bytes and add an async method to the StrompyJsonReader type that yields a PyResult<Option<Vec<Vec<f64>>>> and is exposed with the name 'next'.
If you've done it correctly, you should be able to run it with the following commands:
$ maturin develop
$ python strompy_test_async.py
7.2.2.C: More features (Bonus) ⭐⭐⭐
Can you make strompy support more operations from Nalgebra?
7.2.2.D: Tidying up (Bonus) ⭐⭐⭐⭐
Representing the vectors as Vec<Vec<f64>> works, but it's not great. Make the StrompyJsonReader produce PyResult<Option<Py<PyList>>>s that constists of PyLists instead.
This is quite a hard exercise, as it involves working with PyO3s smart pointers: Py<T>, Bound<'py, T>, and GIL tokens: Python
Unit 8.1 - The Rust embedded ecosystem
Exercise 8.1.1: LSM303AGR ID
Use our newly gained knowledge to get our first application running and read out the ID of the LSM303AGR accelerometer. We can communicate with the LSM303AGR using the I2C that is present on the micro:bit board. Note that the nRF52833 supports I2C with its TWIM (Two-Wire Interface Master) peripheral.
To get started we'll setup the i2c peripheral on our development kit and read out the ID register of the LSM303AGR accelerometer.
The starting point can be found in the file at exercises/8-embedded/1-embedded-ecosystem/src/main.rs in the repository.
Try to run the existing project and then fill in the functionality as instructed by the comments.
To use that project, you can use the following commands from inside that folder using the terminal:
cargo build: Builds the projectcargo run: Builds the project, flashes it to the device and listens for any logs which it will display in the terminal. (This uses theprobe-rs runtool)
In both cases you can add the --release flag to turn on optimizations.
Some pointers to help you get started
- You can find the documentation on the HAL here by running
cargo doc --open. This will build the documentation for the exact chip and version we are using. Normally you'd search at docs.rs, but that may show the HAL for a different NRF variant. - To find out how to configure I2C for the nRF52833: embassy-nrf TWIM demo example.
- You can find the LSM303AGR data sheet here: https://www.st.com/resource/en/datasheet/lsm303agr.pdf. You can find the accelerometer device ID in the
WHO_AM_I_Aregister, at register address0x0F. You'll need0x19to address the accelerometer itself. - Use the
Twim::blocking_write_readmethod to first write the device address, then write the register address, and then read its contents into a buffer. (Note: You can search in your generated docs)
Unit 8.2 - Portable Rust drivers
Exercise 8.2.1: LSM303AGR Driver
When you really want to use a device, you want to have a driver. Let's write an actual portable device driver for the accelerometer we've got.
Go to the assignment in exercises/8-embedded/2-portable-drivers/1-lsm303agr-driver and implement the lsm303agr module.
The goal is to use embedded-hal for our hardware definitions, so try not to use any nRF-specific types in that module.
Use the driver to read data from the sensor in src/main.rs
As a bonus exercise, support reading the magnetometer as well
Unit 8.3 - Async on Embedded
Exercise 8.3.1: Compass
In this exercise, we'll use the lsm303agr driver. Although the documentation doesn't show it, it supports async if you enable its async feature
Have a look at the examples in the lsm303agr-rs repository to get an idea of how to use this driver.
Using the lsm303agr driver, implement a compass. You can use the dial module to indicate the north pole's direction. You'll find a couple of todo!()'s with instructions.
Compiling the starter code yields a bunch of warnings. They'll be gone once your done.
Exercise 8.3.2: Blinky compass
The channel sender - receiver example in the embassy repository shows how to spawn separate tasks, and how to use channels to communicate between tasks. Using that knowledge, make the indicator LED in the dial module blink while magnetometer measurements are taken at the same time.
As we're not using defmt in this exercise, replace the unwrap!(<expr>) macro invocations with <expr>.unwrap() calls.
Unit 8.4 - The Embassy Framework
Exercise 8.4.1: Embassy project
This exercise is a bit bigger than you're used to: we're going to do a custom project using Embassy and the micro:bit V2. Work together in teams of 2 to 3 people during this exercise.
We have set up a crate for you to work on, with a simple blinky application. You're free to adapt anything about it, it's just meant as a starting point. The crate resides in exercises/8-embedded/4-embassy-framework/1-embassy-project. Try it out to ensure the initial setup works.
You can choose from any of below projects:
- Build a snake game that uses the LED matrix as game display. You can use the buttons as input, or use the accelerometer or magnetometer. Or a combination.
- Create a light banner that reads text from UART, using the virtual COM port that is exposed by the interface MCU and shows it on the LED matrix.
- Create an audio recorder that reads samples from the on-board microphone and replays it using the on-board speaker. Additionally, you can show an FFT chart or a volume meter on the display
- Write a driver for the capacitive touch sensor
- Build a button masher multiplayer game. Who is the fastest button presser? Either have it work with the two on-board buttons, or connect multiple boards together somehow. Show the score on the LED matrix.
Wrap-up
Evaluation form
Thank you for taking the time to help us improve the training!
You can find the evaluation from here