Hi.

I am developing bindings for libconfig and discovered a strange bug in =20
irb1.9 (ruby 1.9.0 (2008-06-20 revision 17482) [i486-linux], irb =20
0.9.5(05/04/13); Debian GNU/Linux 5.0/lenny): ruby1.9 executes my =20
script nice, but irb crashes at some random place. Crash is affected =20
by most theoretically unlinked commands and may disappear even after =20
removing a comment (sic).
You can download my extension sources here =20
(http://files.whitequark.ru/libconfig-ruby.tbz2).

Crashing script:
--- BEGIN ---
require 'rconfig'
c =3D Config.new

c.read('test.cfg')
# =3D> false

c.append 'fixnum', Config::Fixnum.new(150)
# #<Config::Fixnum...>

f1 =3D Config::Fixnum.new(1)
c.append 'another_fixnum', f1

f2 =3D Config::Fixnum.new(256)
c.append 'next_fixnum', f2

p c.size
# =3D> 2

c.delete(f1) # by element
c.delete(0) # by index
c.delete('next_fixnum') # by name
# note: (at now) you cannot delete nested elements by Config#delete
# you can do c['nested.element'].parent.delete(c['nested.element'])

p c.size
# =3D> 0

l =3D Config::List.new
c.append 'the_list', l

l.append Config::String.new("abcdef")
l << Config::Float.new(3.14)
--- END ---

Sample log:
--- BEGIN ---
whitequark@hell:~$ tar xf libconfig-ruby.tbz2
whitequark@hell:~$ cd libconfig-ruby/ext/
whitequark@hell:~/libconfig-ruby/ext$ ruby1.9 extconf.rb
checking for rb_block_call() in ruby/ruby.h... yes
creating Makefile
whitequark@hell:~/libconfig-ruby/ext$ make
cc -I. -I/usr/include/ruby-1.9.0/i486-linux -I/usr/include/ruby-1.9.0 =20
-I. -DHAVE_RB_BLOCK_CALL -D_FILE_OFFSET_BITS=3D64 -fPIC =20
-fno-strict-aliasing -g -g -O2 -O2 -g -Wall -Wno-parentheses -fPIC =20
-o rconfig.o -c rconfig.c
cc -shared -o rconfig.so rconfig.o -L. -L/usr/lib -L. -rdynamic =20
-Wl,-export-dynamic -lruby1.9 -lconfig -lpthread -lrt -ldl =20
-lcrypt -lm -lc
whitequark@hell:~/libconfig-ruby/ext$ ruby1.9 crash.rb
3

whitequark@hell:~/libconfig-ruby/ext$ irb1.9 crash.rb
crash.rb(main):001:0> require 'rconfig'
=3D> true
crash.rb(main):002:0> c =3D Config.new
=3D> #<Config:0x9dd76d4 @config=3D#<Object:0x9dd76c0>>
crash.rb(main):003:0>
crash.rb(main):004:0* c.read('test.cfg')
=3D> false
crash.rb(main):005:0> # =3D> false
crash.rb(main):006:0*
crash.rb(main):007:0* c.append 'fixnum', Config::Fixnum.new(150)
=3D> #<Config::Fixnum:0x9dc0dd0 @setting=3D#<Object:0x9dc0d30>, =20
@value=3D150, @format=3D0>
crash.rb(main):008:0> # #<Config::Fixnum...>
crash.rb(main):009:0*
crash.rb(main):010:0* f1 =3D Config::Fixnum.new(1)
=3D> #<Config::Fixnum:0x9db6a9c @setting=3Dnil, @value=3D1, @format=3D0>
crash.rb(main):011:0> c.append 'another_fixnum', f1
=3D> #<Config::Fixnum:0x9db6a9c @setting=3D#<Object:0x9db11f0>, @value=3D1, =
=20
@format=3D0>
crash.rb(main):012:0>
crash.rb(main):013:0* f2 =3D Config::Fixnum.new(256)
=3D> #<Config::Fixnum:0x9daa828 @setting=3Dnil, @value=3D256, @format=3D0>
crash.rb(main):014:0> c.append 'next_fixnum', f2
=3D> #<Config::Fixnum:0x9daa828 @setting=3D#<Object:0x9da86f4>, =20
@value=3D256, @format=3D0>
crash.rb(main):015:0>
crash.rb(main):016:0* p c.size
3
=3D> 3
crash.rb(main):017:0> # =3D> 2
crash.rb(main):018:0*
crash.rb(main):019:0* c.delete(f1) # by element
=3D> nil
crash.rb(main):020:0> c.delete(0) # by index
=3D> nil
crash.rb(main):021:0> c.delete('next_fixnum') # by name
=3D> nil
crash.rb(main):022:0> # note: (at now) you cannot delete nested =20
elements by Config#delete
crash.rb(main):023:0* # you can do =20
c['nested.element'].parent.delete(c['nested.element'])
crash.rb(main):024:0*
crash.rb(main):025:0* p c.size

=3D> 0
crash.rb(main):026:0> # =3D> 0
crash.rb(main):027:0*
crash.rb(main):028:0* l =3D Config::List.new
=3D> #<Config::List:0x9ccb128 @list=3D[], @setting=3Dnil>
crash.rb(main):029:0> c.append 'the_list', l
=3D> #<Config::List:0x9ccb128 @list=3D[], @setting=3D#<Object:0x9cc91fc>>
crash.rb(main):030:0>
crash.rb(main):031:0* l.append Config::String.new("abcdef")
=3D> #<Config::String:0x9c7e92c @setting=3D#<Object:0x9c7e8c8>, =20
@value=3D"abcdef", @format=3D0>
crash.rb(main):032:0> l << Config::Float.new(3.14)
crash.rb:32: [BUG] Segmentation fault
ruby 1.9.0 (2008-06-20 revision 17482) [i486-linux]

-- control frame ----------
c:0024 p:---- s:0080 b:0080 l:000079 d:000079 CFUNC :<<
c:0023 p:0021 s:0076 b:0076 l:001f74 d:0002fc EVAL crash.rb:32
c:0022 p:---- s:0074 b:0074 l:000073 d:000073 FINISH :empty?
c:0021 p:---- s:0072 b:0072 l:000071 d:000071 CFUNC :eval
c:0020 p:0023 s:0065 b:0065 l:000064 d:000064 METHOD =20
/usr/lib/ruby/1.9.0/irb/workspace.rb:80
c:0019 p:0025 s:0058 b:0057 l:000056 d:000056 METHOD =20
/usr/lib/ruby/1.9.0/irb/context.rb:218
c:0018 p:0024 s:0052 b:0052 l:0007bc d:000051 BLOCK =20
/usr/lib/ruby/1.9.0/irb.rb:149
c:0017 p:0025 s:0044 b:0044 l:000043 d:000043 METHOD =20
/usr/lib/ruby/1.9.0/irb.rb:263
c:0016 p:0009 s:0039 b:0039 l:0007bc d:000038 BLOCK =20
/usr/lib/ruby/1.9.0/irb.rb:146
c:0015 p:0093 s:0035 b:0035 l:000022 d:000034 BLOCK =20
/usr/lib/ruby/1.9.0/irb/ruby-lex.rb:243
c:0014 p:---- s:0034 b:0034 l:000033 d:000033 FINISH :block_given?
c:0013 p:---- s:0032 b:0032 l:000031 d:000031 CFUNC :loop
c:0012 p:0007 s:0029 b:0029 l:000022 d:000028 BLOCK =20
/usr/lib/ruby/1.9.0/irb/ruby-lex.rb:230
c:0011 p:---- s:0029 b:0029 l:000028 d:000028 FINISH :each
c:0010 p:---- s:0027 b:0027 l:000026 d:000026 CFUNC :catch
c:0009 p:0017 s:0023 b:0023 l:000022 d:000022 METHOD =20
/usr/lib/ruby/1.9.0/irb/ruby-lex.rb:229
c:0008 p:0034 s:0020 b:0020 l:0007bc d:0007bc METHOD =20
/usr/lib/ruby/1.9.0/irb.rb:145
c:0007 p:0009 s:0017 b:0017 l:001d6c d:000016 BLOCK =20
/usr/lib/ruby/1.9.0/irb.rb:69
c:0006 p:---- s:0017 b:0017 l:000016 d:000016 FINISH :(null)
c:0005 p:---- s:0015 b:0015 l:000014 d:000014 CFUNC :catch
c:0004 p:0152 s:0011 b:0011 l:001d6c d:001d6c METHOD =20
/usr/lib/ruby/1.9.0/irb.rb:68
c:0003 p:0033 s:0006 b:0006 l:000005 d:000005 TOP /usr/bin/irb1.9:12
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH :private_class_method
c:0001 p:0000 s:0002 b:0002 l:000001 d:000001 TOP <dummy toplevel>:17
---------------------------
DBG> : "crash.rb:32:in `<<'"
DBG> : "crash.rb:32:in `irb_binding'"
DBG> : "/usr/lib/ruby/1.9.0/irb/workspace.rb:80:in `eval'"
DBG> : "/usr/lib/ruby/1.9.0/irb/workspace.rb:80:in `evaluate'"
DBG> : "/usr/lib/ruby/1.9.0/irb/context.rb:218:in `evaluate'"
DBG> : "/usr/lib/ruby/1.9.0/irb.rb:149:in `block (2 levels) in eval_input'"
DBG> : "/usr/lib/ruby/1.9.0/irb.rb:263:in `signal_status'"
DBG> : "/usr/lib/ruby/1.9.0/irb.rb:146:in `block in eval_input'"
DBG> : "/usr/lib/ruby/1.9.0/irb/ruby-lex.rb:243:in `block (2 levels) =20
in each_top_level_statement'"
DBG> : "/usr/lib/ruby/1.9.0/irb/ruby-lex.rb:230:in `loop'"
DBG> : "/usr/lib/ruby/1.9.0/irb/ruby-lex.rb:230:in `block in =20
each_top_level_statement'"
DBG> : "/usr/lib/ruby/1.9.0/irb/ruby-lex.rb:229:in `catch'"
DBG> : "/usr/lib/ruby/1.9.0/irb/ruby-lex.rb:229:in `each_top_level_statement=
'"
DBG> : "/usr/lib/ruby/1.9.0/irb.rb:145:in `eval_input'"
DBG> : "/usr/lib/ruby/1.9.0/irb.rb:69:in `block in start'"
DBG> : "/usr/lib/ruby/1.9.0/irb.rb:68:in `catch'"
DBG> : "/usr/lib/ruby/1.9.0/irb.rb:68:in `start'"
DBG> : "/usr/bin/irb1.9:12:in `<main>'"
-- backtrace of native function call (Use addr2line) --
0xb7efac46
0xb7e243a9
0xb7e2441a
0xb7eaba56
0xb7f6e40c
0xb7ee5980
0xb7ef2b07
0xb7ef3a12
0xb7e56f2a
0xb7e013e5
0xb7f61434
0xb7ee6bfb
0xb7ee9b55
0xb7eec59e
0xb7ef1a84
0xb7ef2613
0xb7ef2a4e
0xb7ee6c30
0xb7ee9b55
0xb7eeb8b4
0xb7ef1a84
0xb7ef6add
0xb7e263c1
0xb7ee872c
0xb7ee6c12
0xb7ee9b55
0xb7eeb8b4
0xb7ef1a84
0xb7ef64a9
0xb7ee6c30
0xb7ee9b55
0xb7eeb8b4
0xb7ef1a84
0xb7ef64a9
0xb7ee6c30
0xb7ee9b55
0xb7eeb8b4
0xb7ef1a84
0xb7ef1cad
0xb7e26bd7
0xb7e28156
0x8048794
0xb7c17775
0x8048681
-------------------------------------------------------
--- END ---

On 24.03.2009 14:36, Peter Zotov wrote:
> I am developing bindings for libconfig and discovered a strange bug in
> irb1.9 (ruby 1.9.0 (2008-06-20 revision 17482) [i486-linux], irb
> 0.9.5(05/04/13); Debian GNU/Linux 5.0/lenny): ruby1.9 executes my
> script nice, but irb crashes at some random place. Crash is affected
> by most theoretically unlinked commands and may disappear even after
> removing a comment (sic).

Spontaneously two things come to mind: garbage collection (because of
the seemingless random timing) and real threads (which are introduced in
1.9). Maybe your cleanup code is not thread safe or does not play well
with concurrent GC (not sure whether the 1.9 runtime really does this
but it uses native threads IIRC).

Kind regards

robert

Quoting "Robert Klemme" <shortcutter>:
> Spontaneously two things come to mind: garbage collection (because
> of the seemingless random timing) and real threads (which are
> introduced in 1.9). Maybe your cleanup code is not thread safe or
> does not play well with concurrent GC (not sure whether the 1.9
> runtime really does this but it uses native threads IIRC).

Disabling GC solved this problem. I think only this
Data_Wrap_Struct(rb_cObject, 0, 0, setting)
place of code can be buggy.
I passed NULL to "free" parameter because I do not allocate memory for
setting structure and do not free it (not sure what will be if it is
freed by library, probably I need to make some checks). What do I need
to pass as destructor in this case?

On 24.03.2009 19:09, Peter Zotov wrote:
> Quoting "Robert Klemme" <shortcutter>:
>> Spontaneously two things come to mind: garbage collection (because
>> of the seemingless random timing) and real threads (which are
>> introduced in 1.9). Maybe your cleanup code is not thread safe or
>> does not play well with concurrent GC (not sure whether the 1.9
>> runtime really does this but it uses native threads IIRC).
>
> Disabling GC solved this problem. I think only this
> Data_Wrap_Struct(rb_cObject, 0, 0, setting)
> place of code can be buggy.
> I passed NULL to "free" parameter because I do not allocate memory for
> setting structure and do not free it (not sure what will be if it is
> freed by library, probably I need to make some checks). What do I need
> to pass as destructor in this case?

I have to create a C extension yet, but you certainly need to clearly
separate memory that you allocate in the extension and memory that you
allocate via Ruby's core. The first type needs to be manually freed in
the extension and the latter is probably subject to GC. But someone
else will be able to answer this much better than I can. You could as
well look into another extension and see how they do it there.

HTH

robert

Quoting "Robert Klemme" <shortcutter>:

> On 24.03.2009 19:09, Peter Zotov wrote:
>
> I have to create a C extension yet, but you certainly need to =20
> clearly separate memory that you allocate in the extension and =20
> memory that you allocate via Ruby's core. The first type needs to =20
> be manually freed in the extension and the latter is probably =20
> subject to GC. But someone else will be able to answer this much =20
> better than I can. You could as well look into another extension =20
> and see how they do it there.

The problem is that I do not allocate that memory in _extension_. It =20
is allocated by wrapped library, not me. Okay, I added hooks that =20
notify Ruby objects that setting was freed by library, and changed 0 =20
in "free" parameter of Data_Wrap_Struct to empty destructor.
Nothing happened, same segfault is in same place.

On 24.03.2009 20:07, Peter Zotov wrote:
> Quoting "Robert Klemme" <shortcutter>:
>> The problem is that I do not allocate that memory in _extension_. It
> is allocated by wrapped library, not me. Okay, I added hooks that
> notify Ruby objects that setting was freed by library, and changed 0
> in "free" parameter of Data_Wrap_Struct to empty destructor.
> Nothing happened, same segfault is in same place.

I consider the lib as part of the extension for this example. The
difference is really which malloc is used.

Cheers

robert

Peter, this is bug in your code, not ext lib or ruby-1.9

Actually, your problem is that in your code (rconfig.c) you create
some intermediate objects that *can* be garbage collected in IRB
session (and *will be* under some conditions, as crash.rb shows):

---8<----
aConfigScalars =3D rb_ary_new3(5, cConfigFixnum, cConfigBignum,
cConfigFloat, cConfigBoolean, cConfigString);
aConfigAggregates =3D rb_ary_new3(3, cConfigGroup, cConfigArray, cConfigL=
ist);
aConfigSettings =3D rb_ary_plus(aConfigScalars, aConfigAggregates);
---8<---

as these values are never references from other objects, they die in
GC, receive junk data, and lead to crash:

---8<---
#0 0x081149e0 in search_method (klass=3D105, id=3D325, klassp=3D0x0) at
vm_method.c:229
#1 0x08114a43 in rb_get_method_body (klass=3D105, id=3D325,
idp=3D0xbfffce68) at vm_method.c:256
#2 0x0812074b in rb_call0 (klass=3D105, recv=3D138149640, mid=3D325,
argc=3D1, argv=3D0xbfffce90, scope=3D1, self=3D6) at vm_eval.c:205
#3 0x08120a3f in rb_call () at vm_eval.c:255
#4 rb_funcall (recv=3D138149640, mid=3D325, n=3D1) at vm_eval.c:406
#5 0x0808247d in rb_equal (obj1=3D325, obj2=3D137879300) at object.c:51
#6 0x08132e3d in rb_ary_includes (ary=3D137877820, item=3D137879300) at
array.c:2770
#7 0x005e02b4 in rbConfigAggregate_append (self=3D137015820,
target=3D138267160) at ext/rconfig.c:399
---8<---

usually, in 1.8.x C extensions people overcome this by declaring
global variables, or constants under class (there might be other
options, like freezing, or pure-c managing of consts, but
aforementioned two are most common)

e.g., with such a quick-and-dirty-hack patch, i could not reproduce bug any=
more:
---8<---
--- libconfig-ruby/ext/rconfig.c
+++ libconfig-ruby-mod/ext/rconfig.c
@@ -643,6 +642,9 @@
aConfigScalars =3D rb_ary_new3(5, cConfigFixnum, cConfigBignum,
cConfigFloat, cConfigBoolean, cConfigString);
aConfigAggregates =3D rb_ary_new3(3, cConfigGroup, cConfigArray, cConfig=
List);
aConfigSettings =3D rb_ary_plus(aConfigScalars, aConfigAggregates);
+ rb_define_variable("$RConfigSetting", &aConfigSettings);
+ rb_define_variable("$RConfigScalars", &aConfigScalars);
+ rb_define_variable("$RConfigAggregates", &aConfigAggregates);

char* settingNameRegexp =3D "^[A-Za-z*][A-Za-z\\-_*]*$";
rSettingNameRegexp =3D rb_reg_new(settingNameRegexp,
strlen(settingNameRegexp), 0);
---8<---

two side notes:

1) in ruby-1.9.1 some macros changed (see
http://pennysmalls.com/2009/03/24/ferret-on-ruby-191/ for example):
---8<---
int i;
- for(i =3D 0; i < RARRAY(children)->len; i++) {
- VALUE key =3D RARRAY(children)->ptr[i];
+ for(i =3D 0; i < RARRAY_LEN(children); i++) {
+ VALUE key =3D RARRAY_PTR(children)[i];
rconfig_do_append(new_setting, rb_hash_aref(hash, key), key);
}
}
---8<---

2) just curious, why C extension, and not FFI, as you use quite a
bunch of Ruby methods and few libconfig structs/calls? (see
[url down] )

Thx for this quiz anyway! :)

On Tue, Mar 24, 2009 at 3:36 PM, Peter Zotov <whitequark> wro=
te:
[..]
> c.append 'next_fixnum', f2
>
> p c.size
> # =3D> 2
>
> c.delete(f1) =A0 =A0 =A0 =A0 =A0 =A0# by element
> c.delete(0) =A0 =A0 =A0 =A0 =A0 =A0 # by index
> c.delete('next_fixnum') # by name
> # note: (at now) you cannot delete nested elements by Config#delete
> # =A0 =A0 =A0 you can do c['nested.element'].parent.delete(c['nested.elem=
ent'])
[..]
> Sample log:
> --- BEGIN ---
> whitequark@hell:~$ tar xf libconfig-ruby.tbz2
> whitequark@hell:~$ cd libconfig-ruby/ext/
> whitequark@hell:~/libconfig-ruby/ext$ ruby1.9 extconf.rb
> checking for rb_block_call() in ruby/ruby.h... yes
> creating Makefile
> whitequark@hell:~/libconfig-ruby/ext$ make
> cc -I. -I/usr/include/ruby-1.9.0/i486-linux -I/usr/include/ruby-1.9.0 -I.
> -DHAVE_RB_BLOCK_CALL =A0-D_FILE_OFFSET_BITS=3D64 =A0-fPIC -fno-strict-ali=
asing -g
[..]

Quoting "Dmitry Severin" <dmitry.severin>:

> Peter, this is bug in your code, not ext lib or ruby-1.9
>
> Actually, your problem is that in your code (rconfig.c) you create
> some intermediate objects that *can* be garbage collected in IRB
> session (and *will be* under some conditions, as crash.rb shows):
>
> [skipped]
>
> as these values are never references from other objects, they die in
> GC, receive junk data, and lead to crash:
>

Thanks, fixed everything. Added these arrays as constants.

> 2) just curious, why C extension, and not FFI, as you use quite a
> bunch of Ruby methods and few libconfig structs/calls? (see
> [..] )

Libconfig is a bit Cxx-specific, so I decided to make binding =20
behaviour more rubyish, for example added value caching to allow =20
creation of unattached nodes or deletion of existing (how can I force =20
destroying of Ruby object? :). Also, as I seen, FFI doesn't work on =20
Windows, and that isn't good for me.

If anyone is interested, you can view my final version in repository at

http://git.whitequark.ru/libconfig-ruby.git