C99 designated initializers

One of the very nice features added in C99 is the designated initializer. This allows you to write code like the following to initialize a structure:

    div_t d = { .quot=3, .rem=2 };

In C89 there was no way to reliably initialize a structure like this. The specification says that the quot and rem members may be in any order, so if you write:

    div_t d = { 3, 2 };

you can't be sure which member will be 3 and which will be 2.

In general I'm enamored with designated initializers. But I've run into an unfortunate case where the specification is somewhat ambiguous.

Paragraph §6.7.8.19 of the C99 standard (draft version available for free here) has this to say about the ordering of designated initializers:

18. The initialization shall occur in initializer list order, each initializer provided for a particular subobject overriding any previously listed initializer for the same subobject;¹³⁰ all subobjects that are not initialized explicitly shall be initialized implicitly the same as objects that have static storage duration.
    

(Footnote 130 reads "Any initializer for the subobject which is overridden and so not used to initialize that subobject might not be evaluated at all.")

The spec says that "initialization shall occur in initializer list order". To me, this suggests that one initializer can safely rely on the result of a previous initializer, e.g.:

    div_t d = { .quot=42, .rem=d.quot };

So the following program ought to only invoke well defined behaviour, right?

#include <stdio.h>
#include <stdlib.h>

int main(void) {
    div_t d1 = { .quot=1, .rem=2, };
    printf("d1: quot=%i, rem=%i\n", d1.quot, d1.rem);

    div_t d2 = { .quot=1, .rem=d2.quot };
    printf("d2: quot=%i, rem=%i\n", d2.quot, d2.rem);

    div_t d3 = { .rem=2, .quot=d3.rem };
    printf("d3: quot=%i, rem=%i\n", d3.quot, d3.rem);

    return 0;
}

Let's compile it and see what happens:

18. $ gcc -c99 -O3 -Wall -Wextra foo.c
    $ ./a.out
    d1: quot=1, rem=2
    d2: quot=1, rem=1
    d3: quot=2, rem=2
    
    

Excellent! That exactly what I would expect to happen. The initializers are run in order, allowing the second designated initializer to depend on the result of the first.

Now here's where things start to get weird:

18. $ gcc -c99 -O0 -Wall -Wextra foo.c
    $ ./a.out
    d1: quot=1, rem=2
    d2: quot=1, rem=0
    d3: quot=0, rem=2

If we turn off optimization (-O0) the results suddenly change! Even worse, I've compiled with maximum warnings (-Wall -Wextra) and GCC doesn't even issue a warning about using an uninitialized variable!

How can we reconcile this behaviour with the specification? I think that we need to take paragraph §6.7.8.23 into account, as well:

18. The order in which any side effects occur among the initialization list expressions is unspecified.¹³¹

(Footnote 131 reads "In particular, the evaluation order need not be the same as the order of subobject initialization.")

This suggests that evaluation and initialization are two separate steps: an implementation may evaluate all of the initialization expressions (in any order), record the results in temporary storage, and then apply them all in order. If you inspect the generated assembly code this does seem to match what happens in GCC at -O0.

So why have paragraph 19 at all? I don't see how you could write a C program which can observe the initialization order, except for the special case of one designated initializer overriding another. If that is its only purpose the specification could certainly be more explicit about it. (It's also possible that the specification has actually been clarified in this respect; I don't have access to the final version.)

I've tried this test case with a handful of different compilers and get similar results. However I would be curious to hear about results with other C99 compilers.

Update

It's been a while since I updated this blog. I've been a bit busy, but new articles will start appearing shortly. Since my last post, I have left IBM Canada and joined Two Sigma Investments in New York. I'm no longer developing virtual machines, but the name and scope of the blog will remain the same. I'm still doing low-level software development and I'm learning a lot about areas I haven't investigated in depth before, and from my new colleagues.

IBM Java 7 is now available

IBM Java 7 was officially released yesterday, September 19. You can download it from IBM DeveloperWorks.

There are a lot of exciting new features including a number of GC improvements.

• The balanced GC policy, which I've mentioned before, is included in all 64-bit Java 7 JDKs. You can enable it with -Xgcpolicy:balanced.
• The soft-realtime garbage collector is included for evaluation on Linux and AIX. It can be enabled with -Xgcpolicy:metronome.
• The verbose GC format has been completely overhauled. It now provides more information and the XML format has been redesigned to make machine interpretation of the data simpler, allowing both IBM and customers to write tools to process and analyse the data.

"Don't do what Donny Don't does"

Thank to Evan Hughes for pointing out this paper: Conditional statements, looping constructs, and program comprehension: an experimental study.

Not surprisingly, negative conditions are more difficult to understand than positive conditions.

I always try to write conditions to be positive. Sometimes, I'll even include an empty 'if' block so that I can put code in the 'else' block instead of using a negative condition:

   if (isInRange(value)) {

      // expected case; do nothing

   } else {

      throw new OutOfRangeException(value);

   }

I haven't read the full paper, so maybe the researchers answered my next question: is the problem exacerbated by the syntax for negative conditions used in C-like languages? I find that the '!' operator uses very little horizontal space, making it less noticeable than other unary operators such as '~' or '*'.

e.g. would this statement:

   if (!isInRange(value)) { ...

be more obvious if it were written like this?

   if (not isInRange(value)) { ...

Balanced garbage collection

We've just published an article on the IBM DeveloperWorks website describing the new garbage collection technology available in IBM Java 6 2.6 and IBM Java 7. This is a project I've been working on for several years and we're pretty excited that customers can now try it out for themselves.

You can read the article for yourself.

Reachability follow up

We've been having quite an interesting series of conversations at work about this reachability problem. Today, one of my co-workers, Dan Heidinga, pointed out that Microsoft's CLR has the same issue. For CLR Microsoft has provided a special static method called GC.KeepAlive(Object). This acts as a hint to the virtual machine and JIT to extend an object's lifetime, but is otherwise a no-op. There's a good article about the problem on an old MSDN blog here. Note that the author considers and rejects the option of automatically extending the lifetime of all function arguments to the end of their functions on the basis that it impacts codegen and therefore performance.

A subtle issue of reachability

In the last few weeks I've run into two similar and very subtle problems in Java code. In some ways, these seem to illustrate an oversight in the design the Java language and/or virtual machine. The problem has to do with objects being collected earlier than expected.

In one case a finalizer was run and in the other a PhantomReference was cleared. I'll describe an example based on finalization. It's easy enough to see how this could also apply to reference objects.

Consider a class like this:

public class Foo {

  private byte[] array = new byte[] { 1, 2, 3 };

  public void finalize() { 

    array[0] = array[1] = 

      array[2] = 0; 

  }

  public byte[] getData() { return array.clone(); }

}

This class is able to return a copy of its data array. (This is a common pattern since it prevents the caller from modifying the master copy of the array). When instances of this class are garbage collected they wipe out the data in the array, overwriting it with zeros. (Let's ignore why it does this; it's just an example!) So far so good.

What result will we get if we invoke getData() on an instance of Foo?

  Foo f = new Foo();

  byte[] array = f.getData();

  System.out.println("array={" + 

    array[0] + ", " + 

    array[1] + ", " + 

    array[2] + "}");

Intuitively, we expect this to print "array={1, 2, 3}". And it usually does. But is it legitimate for it to print "array={0, 0, 0}" (or even "array={1, 2, 0}")? If it did, that would mean that the object was finalized while we were still using it, wouldn't it?

Actually, that can happen. It happens quite often in the IBM Java VM, and seems to happen occasionally in Oracle HotSpot, too, but less frequently.

The Java VM is permitted to collect (or finalize) an object when it is no longer reachable. But how could the object become unreachable when we're running the getData() function? It's easier to understand if you imagine the function in-lined in the caller, and broken up into individual statements:

  Foo f = new Foo();

  byte[] masterArray = f.array; // ignore that array
                                // is private

  // what if a garbage collection happens here?
  // e.g. System.gc(); System.runFinalization();

  byte[] copyArray = masterArray.clone();

  System.out.println("array={" + 

    copyArray[0] + ", " + 

    copyArray[1] + ", " + 

    copyArray[2] + "}");

Here we can see that if the garbage collector interrupts the program at just the right (wrong?) time, the finalize() function might run before we clone the array. Even though we don't explicitly assign null to f, a clever VM can analyze the program and determine that f is never used again. It can reclaim the memory for that object and, in this case, finalize it, before the clone() function runs. In most cases this is exactly what you want the VM to do: garbage collect objects as early as possible to recover as much memory as possible.

Ok, but is that really the same thing? Surely, the receiver of a function is kept alive until the function returns, right? In-lining the function isn't quite the same!

Actually, neither the Java language specification nor the Java Virtual Machine specification say anything about that. In the VM, the receiver of a function (i.e. this) isn't very special at all. It's just the first argument of a virtual function. Although the language doesn't allow it (keep in mind that the Java language and the Java VM have separate specifications) you can overwrite the receiver just as you can a local variable if you're writing bytecodes directly without the aid of javac:

byte[] getData() {

  byte[] masterArray = this.array;

  this = null; // not legal in Java language,
               // but is legal in class files!

  return masterArray.clone();

}

javac won't compile this, but the JVM's class file verifier won't report any problems in this function.

So, what's the right way to write your Java code so that your objects won't be finalized or collected earlier than expected? Unfortunately, I don't know the answer. You could add an extra reference to the receiver, like this:

byte[] getData() {

  byte[] result = array.clone();

  this.array = this.array;

  return result;

}

But that's a hack, not a real solution, and is unlikely to work reliably. The VM can easily determine that the dummy "this.array = this.array" statement has no effect and can be removed, leaving us exactly where we started.

Perhaps Java needs a new keyword like this:

byte[] getData() {

  keep_alive(this) {

    return array.clone();

  }

}

However I doubt that something like that would be used correctly very often.

Unfortunately, the best advice is probably to avoid finalization whenever possible.