Benchmarking of tebako package against original Ruby applications

Maxim Samsonov on 23 Nov 2023

Rationale

Tebako is an executable packager. It packages a set of files into a single executable binary that allows a user to run a selected file from the packaged software as if it is a mounted filesystem.

As we discussed in the earlier post tebako introduces additional layer for most of functions tha access filesystem. This layer routes calls either to original implementation or to dwarfs in-memory packaged filesystem. If a call is served by dwarfs tebako consumes additional resources for decompression.

If Ruby application includes native extension, its shared library is extracted to host temporarily folder and loaded from there that may create addional delay.

Finally Tebako package includes the whole Ruby standard library and all files that were used to install and build additional Ruby gems and theier native extension. The bigger file may taje additional time for startup.

These four features may have performance impact that we have estimated and discuss in the present post.

Executive summary

In our very moderate test environment (specified below) we observed the following behavior:

• Tebako package parses Ruby code faster then native interpreter. Routing from additional file system access layer + reading dwarfs including decompression is faster then reading from disk, even from SSD.

• External file processing is not noticebly slower for Tebako package despite of additional file system access layer introduced by Tebako

• Tebako package initialization adds a 'penalty' time needed to read a fairly big file that conatains Ruby standard lib. The application not necessarily uses the whole library but we still need to map it to application memory address space on startup.

• Any native extension or other native code like Java jars in tebako add initialization time because we are decompressing them to temp and loading again

Tebako package introduces additional time that is consumed during initialization. This time depends on the application size and the size of its native extensions, it dows not depend on complexity of the data processed by the application or any other volatile factors In our tests additional time mentioned above varies from 0.03 seconds for 'Hello, world" script to 3 seconds for metanorma application

Tests and results

In order to address different aspects of possible performance issues we ran 4 different tests:

• Simple "Hello, world!" script that allowes to measure performance of Ruby core

• Coradoc gem that uses Ruby code to process external files

• Vectory gem that uses native extension to process external filesystem

• Сomplex metanorma application that combines uses numerous Ruby gems, native extensions and Jave code

Simple sript and coradoc, vectory gem tests included several runs with varying repetition of calls. Metanorma test included execution of various commands that generate various load profiles.

'Hello, world!' scripts

if (argv = ARGV).empty?
  puts "No arguments given"
  exit(1)
end

if argv[0].to_i < 1
  puts "Argument must be a positive integer"
  exit(1)
end

argv[0].to_i.times do |i|
  puts "Hello, world number #{i}!"
  puts "Gem path: #{Gem.path}"
end

Coradoc gem

if (argv = ARGV).empty?
  puts "No arguments given"
  exit(1)
end

if argv[0].to_i < 1
  puts "Argument must be a positive integer"
  exit(1)
end

argv[0].to_i.times do
  require "coradoc"
  sample_file = File.join(__dir__, "fixtures", "sample.adoc")
  require "coradoc/legacy_parser"
  Coradoc::LegacyParser.parse(sample_file)[:document]

  require "coradoc/oscal"
  sample_file = File.join(__dir__, "fixtures", "sample-oscal.adoc")
  document = Coradoc::Document.from_adoc(sample_file)
  Coradoc::Oscal.to_oscal(document)

  syntax_tree = Coradoc::Parser.parse(sample_file)
  Coradoc::Transformer.transform(syntax_tree)
end

Vectory gem benchmarks

require "tempfile"

if (argv = ARGV).empty?
  puts "No arguments given"
  exit(1)
end

if argv[0].to_i < 1
  puts "Argument must be a positive integer"
  exit(1)
end

argv[0].to_i.times do
  require "emf2svg"

  svg = Emf2svg.from_file(File.join(__dir__, "fixtures", "img.emf"))

  Tempfile.create(["output", ".svg"]) do |tempfile|
    tempfile.write(svg)
    puts "SVG written to #{tempfile.path}"
  end
end

Metanorma package benchmarking

Metanorma application benchmarking included execution of utility commands: metanorma help, metanorma version and generation of sample sites (ietf, ieee, iec, iso, iho) using

metanorma site generate samples -c samples/metanorma.yml  -o site-<site name> --agree-to-terms

Benchmarking environment

Model Name:	Mac mini
Model Identifier:	Macmini9,1
Chip:	Apple M1
Total Number of Cores:	8 (4 performance and 4 efficiency)
Memory:	16 GB
Ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [arm64-darwin21]
tebako executable packager 0.5.5