Gems with Extensions

Creating a gem that includes an extension that is built at install time.

Many gems use extensions to wrap libraries that are written in C with a ruby wrapper. Examples include nokogiri which wraps libxml2 and libxslt, pg which is an interface to the PostgreSQL database and the mysql and mysql2 gems which provide an interface to the MySQL database.

Creating a gem that uses an extension involves several steps. This guide will focus on what you should put in your gem specification to make this as easy and maintainable as possible. The extension in this guide will wrap malloc() and free() from the C standard library.

Extensions don’t have to be written in C. This guide uses C for its main example, but you can also write extensions in Rust; see the Rust Extensions section below.

Gem layout

Every gem should start with a Rakefile which contains the tasks needed by developers to work on the gem. The files for the extension should go in the ext/ directory in a directory matching the extension’s name. For this example we’ll use “my_malloc” for the name.

Some extensions will be partially written in C and partially written in ruby. If you are going to support multiple languages, such as C and Java extensions, you should put the C-specific ruby files under the ext/ directory as well in a lib/ directory.

Rakefile
ext/my_malloc/extconf.rb               # extension configuration
ext/my_malloc/my_malloc.c              # extension source
lib/my_malloc.rb                       # generic features

When the extension is built the files in ext/my_malloc/lib/ will be installed into the lib/ directory for you.

extconf.rb

The extconf.rb configures a Makefile that will build your extension. The extconf.rb must check for the necessary functions, macros and shared libraries your extension depends upon. The extconf.rb must exit with an error if any of these are missing.

Here is an extconf.rb that checks for malloc() and free() and creates a Makefile that will install the built extension at lib/my_malloc/my_malloc.so:

require "mkmf"

abort "missing malloc()" unless have_func "malloc"
abort "missing free()"   unless have_func "free"

create_makefile "my_malloc/my_malloc"

See the mkmf documentation and extension.rdoc for further information about creating an extconf.rb and for documentation on these methods.

C Extension

The C extension that wraps malloc() and free() goes in ext/my_malloc/my_malloc.c. Here’s the listing:

#include <ruby.h>

struct my_malloc {
  size_t size;
  void *ptr;
};

static void
my_malloc_free(void *p) {
  struct my_malloc *ptr = p;

  if (ptr->size > 0)
    free(ptr->ptr);
}

static VALUE
my_malloc_alloc(VALUE klass) {
  VALUE obj;
  struct my_malloc *ptr;

  obj = Data_Make_Struct(klass, struct my_malloc, NULL, my_malloc_free, ptr);

  ptr->size = 0;
  ptr->ptr  = NULL;

  return obj;
}

static VALUE
my_malloc_init(VALUE self, VALUE size) {
  struct my_malloc *ptr;
  size_t requested = NUM2SIZET(size);

  if (0 == requested)
    rb_raise(rb_eArgError, "unable to allocate 0 bytes");

  Data_Get_Struct(self, struct my_malloc, ptr);

  ptr->ptr = malloc(requested);

  if (NULL == ptr->ptr)
    rb_raise(rb_eNoMemError, "unable to allocate %ld bytes", requested);

  ptr->size = requested;

  return self;
}

static VALUE
my_malloc_release(VALUE self) {
  struct my_malloc *ptr;

  Data_Get_Struct(self, struct my_malloc, ptr);

  if (0 == ptr->size)
    return self;

  ptr->size = 0;
  free(ptr->ptr);

  return self;
}

void
Init_my_malloc(void) {
  VALUE cMyMalloc;

  cMyMalloc = rb_const_get(rb_cObject, rb_intern("MyMalloc"));

  rb_define_alloc_func(cMyMalloc, my_malloc_alloc);
  rb_define_method(cMyMalloc, "initialize", my_malloc_init, 1);
  rb_define_method(cMyMalloc, "free", my_malloc_release, 0);
}

This extension is simple with just a few parts:

struct my_malloc to hold the allocated memory
my_malloc_free() to free the allocated memory after garbage collection
my_malloc_alloc() to create the ruby wrapper object
my_malloc_init() to allocate memory from ruby
my_malloc_release() to free memory from ruby
Init_my_malloc() to register the functions in the MyMalloc class.

Now we can create the actual MyMalloc class and bind newly defined methods in Ruby (lib/my_malloc.rb is the correct place for that), e.g.:

class MyMalloc
  VERSION = "1.0"
end

require "my_malloc/my_malloc"

You can test building the extension as follows:

$ cd ext/my_malloc
$ ruby extconf.rb
checking for malloc()... yes
checking for free()... yes
creating Makefile
$ make
compiling my_malloc.c
linking shared-object my_malloc.bundle
$ cd ../..
$ ruby -Ilib:ext -r my_malloc -e "p MyMalloc.new(5).free"
#<MyMalloc:0x007fed838addb0>

But this will get tedious after a while. Let’s automate it!

rake-compiler

rake-compiler is a set of rake tasks for automating extension building. rake-compiler can be used with C or Java extensions in the same project (nokogiri uses it this way).

First install the gem:

$ gem install rake-compiler

Adding rake-compiler to the Rakefile is very simple:

require "rake/extensiontask"

Rake::ExtensionTask.new "my_malloc" do |ext|
  ext.lib_dir = "lib/my_malloc"
end

Now you can build the extension with rake compile and hook the compile task into other tasks (such as tests).

Setting lib_dir places the shared library in lib/my_malloc/my_malloc.so (or .bundle or .dll). This allows the top-level file for the gem to be a ruby file. This allows you to write the parts that are best suited to ruby in ruby.

For example:

class MyMalloc

  VERSION = "1.0"

end

require "my_malloc/my_malloc"

Setting the lib_dir also allows you to build a gem that contains pre-built extensions for multiple versions of ruby. (An extension for Ruby 1.9.3 cannot be used with an extension for Ruby 2.0.0). lib/my_malloc.rb can pick the correct shared library to install.

Gem specification

The final step to building the gem is adding the extconf.rb to the extensions list in the gemspec:

Gem::Specification.new "my_malloc", "1.0" do |s|
  # [...]

  s.extensions = %w[ext/my_malloc/extconf.rb]
end

Now you can build and release the gem!

Extension Naming

To avoid unintended interactions between gems, it’s a good idea for each gem to keep all of its files in a single directory. Here are the recommendations for a gem with the name <name>:

ext/<name> is the directory that contains the source files and extconf.rb
ext/<name>/<name>.c is the main source file (there may be others)
ext/<name>/<name>.c contains a function Init_<name>. (The name following Init_ function must exactly match the name of the extension for it to be loadable by require.)
ext/<name>/extconf.rb calls create_makefile('<name>/<name>') only when the all the pieces needed to compile the extension are present.
The gemspec sets extensions = ['ext/<name>/extconf.rb'] and includes any of the necessary extension source files in the files list.
lib/<name>.rb contains require '<name>/<name>' which loads the C extension

Rust Extensions

Native extensions can also be written in Rust instead of C. Since RubyGems 3.3.11 a gem can declare a Rust extension that is compiled at install time, giving you Rust’s memory safety and the Cargo build system while still producing an ordinary shared library that Ruby loads with require.

The recommended toolchain is maintained by the oxidize-rb project:

rb-sys wires Cargo into the standard rake-compiler workflow, so building, testing, and cross-compiling work the same way they do for C extensions.
magnus provides a high-level, safe API for defining Ruby modules, classes, and methods from Rust. You can drop down to the raw rb-sys bindings when you need to, but most gems should use magnus.

You’ll need a Rust toolchain installed in addition to Ruby.

Generating the gem

The quickest way to start is to let Bundler scaffold the whole project:

$ bundle gem --ext=rust my_gem

This produces a working gem with all of the Rust wiring already in place:

Cargo.toml                         # Cargo workspace
Rakefile
my_gem.gemspec
ext/my_gem/Cargo.toml              # crate definition and dependencies
ext/my_gem/extconf.rb              # extension configuration
ext/my_gem/src/lib.rs              # extension source
lib/my_gem.rb                      # generic features

The rest of this section walks through the pieces that differ from a C extension.

extconf.rb

Instead of mkmf’s create_makefile, a Rust extension uses create_rust_makefile from rb_sys/mkmf. It generates a Makefile that drives Cargo:

require "mkmf"
require "rb_sys/mkmf"

create_rust_makefile("my_gem/my_gem")

The argument is the install path of the compiled library, exactly like the argument to create_makefile. Here it places the shared object at lib/my_gem/my_gem.so.

Cargo.toml

Each extension is a Cargo crate. The two things that matter for a Ruby extension are the cdylib crate type, so Cargo produces a shared library Ruby can load, and the magnus dependency:

[package]
name = "my_gem"
version = "0.1.0"
edition = "2021"
publish = false

[lib]
crate-type = ["cdylib"]

[dependencies]
magnus = "0.8"

A Cargo.toml in the project root declares a workspace so that editors and cargo commands run from the top of the gem behave correctly:

[workspace]
members = ["./ext/my_gem"]
resolver = "2"

The Rust source

ext/my_gem/src/lib.rs defines the extension. The function marked with #[magnus::init] is called when Ruby loads the library, and it’s where you define your modules, classes, and methods:

use magnus::{function, prelude::*, Error, Ruby};

fn hello(subject: String) -> String {
    format!("Hello {subject}, from Rust!")
}

#[magnus::init]
fn init(ruby: &Ruby) -> Result<(), Error> {
    let module = ruby.define_module("MyGem")?;
    module.define_singleton_method("hello", function!(hello, 1))?;
    Ok(())
}

magnus converts between Ruby and Rust types for you, so hello takes and returns an ordinary Rust String. Calling MyGem.hello("world") from Ruby returns "Hello world, from Rust!".

Rakefile

rb_sys ships a drop-in replacement for Rake::ExtensionTask that knows how to drive Cargo. Use RbSys::ExtensionTask and it hooks into rake compile just like the C workflow:

require "rb_sys/extensiontask"

task build: :compile

GEMSPEC = Gem::Specification.load("my_gem.gemspec")

RbSys::ExtensionTask.new("my_gem", GEMSPEC) do |ext|
  ext.lib_dir = "lib/my_gem"
end

task default: %i[compile test]

lib/my_gem.rb

The top-level Ruby file requires the compiled library, just like a C extension:

require_relative "my_gem/version"
require "my_gem/my_gem"

module MyGem
  class Error < StandardError; end
  # Your code goes here...
end

Gem specification

Point extensions at the extconf.rb and add a dependency on rb_sys:

Gem::Specification.new do |spec|
  spec.name = "my_gem"
  spec.version = MyGem::VERSION
  # [...]

  spec.extensions = ["ext/my_gem/extconf.rb"]
  spec.add_dependency "rb_sys", ">= 0.9.128"
end

Make sure the gemspec’s files list includes the Rust sources and the Cargo.toml files (for example ext/**/*.rs and **/Cargo.*) so they ship in the packaged gem. The scaffold generated by bundle gem already does this via git ls-files.

Building and testing locally

Compile the extension and try it out:

$ bundle install
$ bundle exec rake compile
$ bundle exec ruby -Ilib -r my_gem -e "puts MyGem.hello('world')"
Hello world, from Rust!

rake compile builds the crate with Cargo and copies the resulting shared library into lib/my_gem/, where lib/my_gem.rb can require it. From here rake test works exactly as it would for a C extension.

Although Cargo omits Cargo.lock from version control for library crates by default, you should commit it for a gem. The native extension is built from source on the user’s machine at install time, so a checked-in Cargo.lock gives everyone the same, reproducible set of Rust dependencies. The .gitignore generated by bundle gem ignores only the target/ build directory, leaving Cargo.lock tracked.

Bundler

Gem layout

extconf.rb

C Extension

rake-compiler

Gem specification

Extension Naming

Rust Extensions

Generating the gem

extconf.rb

Cargo.toml

The Rust source

Rakefile

lib/my_gem.rb

Gem specification

Building and testing locally

Further Reading