RTEMS looks interesting, and has many positive points... small, GPL'd, many processors, Linux simulator support. RTEMS 4.0 has a TCP/IP implementation based on KA9Q.
On the other hand, not many people work on it.
Mach is fairly small-- the microkernel anyway-- and has been around for awhile... lots of people work on it. Mach might be a better match. [Note: check JOSWiki for info on how RTEMS was selected as kernel for JOS. What were considerations?][here's the link to them, good discussion: http://www.spin.de/osproject/archive/jos-kernel-archive/ ]
Mach has Linux emulation, and soon will be able to use Linux device drivers...
U. of Utah maintains Mach, also develops Flux, successor to Mach.
OSKit is cool! U. of Utah's OSKit project bolted Kaffe to OSKit and made a microkernel with just java running on it... here's the link...
JavaOS using OSKit: http://www.cs.utah.edu/projects/flexmach/oskit/javaos.html
OSKit paper (including JavaOS-like system description): ftp://mancos.cs.utah.edu/papers/oskit-sosp97.html
A lot of OSKit's kernel functionality is in C, but then, so is RTEMS's, and RTEMS's code isn't nearly as stable and living. Since OSKit's code comes from FreeUnixes, lots of people still maintain it... plus, the fact that UofU could bolt together a JavaOS system in days using OSKit is enough for me!
Problem: OSKit is not currently available, UofU withdrew the old OSKit version in anticipation of a new release.
Linux has java executable support built in; what is minimal linux kernel with java or kaffe installed...?
Ooops! A little research indicates OSKit 0.60 is located here: ftp://ftp.dcrl.nd.edu/pub/OSKit/ Yes!