Making RSpec CI build green for TruffleRuby

Today I will show what problems I found during the work on adding TruffleRuby support for the RSpec project.

RSpec - it's a powerful and popular test framework, which most of you probably already heard about or even used before. The first time I tried to run it on TruffleRuby it failed in 23 test cases. Usually, in such cases I either file an issue in the TruffleRuby repository or, if there are a lot of failures, I skip the gem. So after a year or so I decided to look at the issues closer.

For those who are not familiar with TruffleRuby is a drop-in Ruby implementation aiming to be the fastest and compatible. Even though the TruffleRuby team is constantly working on narrowing the gap between Ruby and TruffleRuby, sometimes it's still needed to update the library codebase to make it work on all versions of Ruby.

After updating around 100 gems I've encountered dozens of interesting Ruby features and generally interesting bugs, which in other circumstances I would never learn about. Starting from today I decided to blog about my journey

Cleaning up TruffleRuby stack trace

First subset of failures was about additional lines in the backtrace. For example here:‌

expected: ...
got: ...

(compared using ==)

Diff:
@@ -5,6 +5,7 @@
RuntimeError:
boom
# ./helper_with_error.rb:1:in `<top (required)>'
+# <internal:core> core/kernel.rb:234:in `gem_original_require'
No examples found.

Essentially the problem is that TruffleRuby added here extran line:‌

+# <internal:core> core/kernel.rb:234:in `gem_original_require'

In general, I would not recommend writing such tests, which expect some particular output, because it's too fragile and will need to be modified in the future if, for example, error formatting will change based on language implementation. To me, it's enough to test that output has some particular substring in it.

My plan was to keep tests changes to a minimum, I had to clean up a bit of backtrace which TruffleRuby was generating to make it equal to what MRI Ruby would produce. It took me a while to find a place where the actual stack trace is caught and formatted. Eventually, I figured out that it's done in another Rspec project, called rspec-support, so the needed change looks like this:

--- a/lib/rspec/support/caller_filter.rb
+++ b/lib/rspec/support/caller_filter.rb
@@ -27,10 +27,6 @@ module RSpec
 
    LANGUAGE_SPECIFIC = []
    LANGUAGE_SPECIFIC <<  "<internal:" if RUBY_ENGINE == 'truffleruby'
    
    IGNORE_REGEX = Regexp.union(LIB_REGEX, "rubygems/core_ext/kernel_require.rb", *LANGUAGE_SPECIFIC)

If ruby engine is truffleruby we will need to clean up backtraces a bit, so it will look like the one which MRI Ruby generates. In particular all lines starting from <internal: will be cleaned up.

For the other similar issues extending of the backtrace_exclusion_patterns usually was enough, so the fix would look like this:

  RSpec.configure do |c|
      ...
      c.backtrace_exclusion_patterns << %r{mri/bundler} if 'truffleruby' == RUBY_ENGINE
      ...
  end

These changes I applied for spec_file_load_errors_spec.rb and suite_hooks_errors_spec.rb

For some cases Dir[] didn't follow the folders which were the symbolic links. To fix it I opened an issue which the TruffleRuby team quickly picked up and fixed.

Side note: shout out to the TruffleRuby team who've been helping me with fixing compatibility issue on the TruffleRuby side.

Other small changes

Another test that was failing in the rspec-support repository was the one related to ruby's -w parameter which turns on the warnings for the script.

For example this small program emits a warning that the field is not initialiezed:

ruby -w -e'puts $undefined'
-e:1: warning: global variable `$undefined' not initialized

without -w it will not be shown

Since this feature is not yet implemented in the TruffleRuby, I simply ignored the failing test

Conclusion

Overall it took me about 2 days to make all tests green. Even if the number of failing tests was terrifying in the beginning, fixing them was quite easy once I've understood that problem. For the harder cases, when a feature is missing in the TruffleRuby itself there is no other way than to file an issue for the TruffleRuby team