Hacking Up an RGB Framebuffer Driver for Wii-Linux

The Hacks – More ‘Real’ RGB565

Basically, my solution is to create a virtual, memory-based RGB framebuffer in the gcnfb driver, acting like a proxy between the actual YUY2 framebuffer and all other programs trying to access it. The actual YUY2 framebuffer will be only accessed from within the driver, but the virtual framebuffer will be exposed to everyone else for read and write. Programs trying to access framebuffer for display will think they are interacting with a normal RGB framebuffer (and they are), which they natively support. RGB-YUY2 conversions will be performed only within the driver and only for writes to the actual framebuffer, because nobody needs the YUY2 data when there are corresponding RGB data available in the virtual framebuffer. There is nothing innovative about this approach. Numerous display devices have been using schemes like this for years, but they usually perform the format conversions in hardware.

To implement the virtual RGB framebuffer without reinventing the wheel, appropriate code from the memory-based virtual framebuffer driver (‘vfb’) in the linux kernel is directly borrowed and used. First, we allocate memory the size of the actual framebuffer in gcnfb.c and point gcnfb’s framebuffer pointers to the allocated memory.

@@ -1872,6 +1898,38 @@ static int vifb_mmap(struct fb_info *inf
 	return 0;

+static int vfb_mmap(struct fb_info *info,
+		    struct vm_area_struct *vma)
+	unsigned long start = vma->vm_start;
+	unsigned long size = vma->vm_end - vma->vm_start;
+	unsigned long offset = vma->vm_pgoff << PAGE_SHIFT; +	unsigned long page, pos; + +	if (offset + size > info->fix.smem_len) {
+		return -EINVAL;
+	}
+	pos = (unsigned long)info->fix.smem_start + offset;
+	while (size > 0) {
+		page = vmalloc_to_pfn((void *)pos);
+		if (remap_pfn_range(vma, start, page, PAGE_SIZE, PAGE_SHARED)) {
+			return -EAGAIN;
+		}
+		start += PAGE_SIZE;
+		pos += PAGE_SIZE;
+		if (size > PAGE_SIZE)
+			size -= PAGE_SIZE;
+		else
+			size = 0;
+	}
+	vma->vm_flags |= VM_RESERVED;	/* avoid to swap out this VMA */
+	return 0;
 static int vifb_ioctl(struct fb_info *info,
 		       unsigned int cmd, unsigned long arg)
@@ -1936,11 +1994,10 @@ static int vifb_ioctl(struct fb_info *in
 struct fb_ops vifb_ops = {
 	.owner = THIS_MODULE,
 	.fb_setcolreg = vifb_setcolreg,
-	.fb_pan_display = vifb_pan_display,
 	.fb_ioctl = vifb_ioctl,
 	.fb_set_par = vifb_set_par,
 	.fb_check_var = vifb_check_var,
-	.fb_mmap = vifb_mmap,
+	.fb_mmap = vfb_mmap,
 	.fb_fillrect = cfb_fillrect,
 	.fb_copyarea = cfb_copyarea,
 	.fb_imageblit = cfb_imageblit,
@@ -1975,29 +2032,56 @@ static int __devinit vifb_do_probe(struc
 	ctl->io_base = ioremap(mem->start, mem->end - mem->start + 1);
 	ctl->irq = irq;

+	void *vfb_mem;
+	unsigned long adr;
+	unsigned long size = PAGE_ALIGN(xfb_size);
+	vfb_mem = vmalloc_32(size);
+	if (!vfb_mem) {
+		drv_printk(KERN_ERR, "failed to allocate virtual framebuffer\n");
+		error = -ENOMEM;
+		goto err_framebuffer_alloc;
+	}
+        else {
+		memset(vfb_mem, 0, size);
+		adr = (unsigned long)vfb_mem;
+		while (size > 0) {
+			SetPageReserved(vmalloc_to_page((void *)adr));
+			adr += PAGE_SIZE;
+			size -= PAGE_SIZE;
+		}
+		drv_printk(KERN_INFO,
+			   "virtual framebuffer at 0x%p, size %dk\n",
+			   vfb_mem, PAGE_ALIGN(xfb_size) / 1024);
+	}
 	 * Location and size of the external framebuffer.
-	info->fix.smem_start = xfb_start;
+	info->fix.smem_start = (unsigned long) vfb_mem;
 	info->fix.smem_len = xfb_size;

-	if (!request_mem_region(info->fix.smem_start, info->fix.smem_len,
+	if (!request_mem_region(xfb_start, xfb_size,
 			   "failed to request video memory at %p\n",
-			   (void *)info->fix.smem_start);
+			   (void *)xfb_start);

-	info->screen_base = ioremap(info->fix.smem_start, info->fix.smem_len);
-	if (!info->screen_base) {
+	/* Save the physical fb info */
+	fb_start = xfb_start;
+	fb_size = xfb_size;
+	fb_mem = ioremap(fb_start, fb_size);
+	if (!fb_mem) {
 			   "failed to ioremap video memory at %p (%dk)\n",
-			   (void *)info->fix.smem_start,
+			   (void *)fb_start,
 			   info->fix.smem_len / 1024);
 		error = -EIO;
 		goto err_ioremap;

+	info->screen_base = (char __iomem *)vfb_mem;

We could in fact allocate only half the size of actual framebuffer and save a maximum of 720K memory, because the physical framebuffer is actually a double buffer. However, later we will see how this 720K could be put to another interesting use.

Next, we have to decide when the RGB565-YUY2 conversion should take place. We obviously shouldn’t do it while data in the physical framebuffer is being read out for display, otherwise we could write over the previous frame before it is fully displayed and cause ‘tearing’. The only viable option is to do it during the vertical retrace, when the previous frame has been fully displayed and scanning of the next frame hasn’t started. To do this, we hook the RGB-YUY2 conversion code in gcnfb.c, which is used originally only by fbcon, with the vtrace interrupt handler instead, and perform a full screen conversion at each vertical retrace, reading from the virtual RGB framebuffer and writing into Wii’s real YUY2 framebuffer.

@@ -585,8 +593,9 @@ static inline void gcngx_dispatch_vtrace
  * Converts two 16bpp rgb pixels into a dual yuy2 pixel.
-static inline uint32_t rgbrgb16toycbycr(uint16_t rgb1, uint16_t rgb2)
+static inline uint32_t rgbrgb16toycbycr(uint32_t rgb1rgb2)
+	uint16_t rgb1 = rgb1rgb2 >> 16, rgb2 = rgb1rgb2 & 0xFFFF;
 	register int Y1, Cb, Y2, Cr;
 	register int r1, g1, b1;
 	register int r2, g2, b2;
@@ -1241,6 +1250,22 @@ static void vi_dispatch_vtrace(struct vi
 	unsigned long flags;

+	/* Copy and convert contents of virtual framebuffer */
+	struct fb_info *info = ctl->info;
+	unsigned int width = info->fix.line_length >> 2;
+	unsigned int height = info->var.yres;
+	uint32_t *addr0 = (uint32_t *)info->screen_base;
+	uint32_t *addr1 = fb_mem;
+	while (height--) {
+		int j = width;
+		while (j--) {
+			*(addr1 + j) = rgbrgb16toycbycr(*(addr0 + j));
+		}
+		addr0 += width;
+		addr1 += width;
+	}
 	spin_lock_irqsave(&ctl->lock, flags);
 	if (ctl->flip_pending)

By now, we already have a working virtual RGB565 framebuffer driver. (Some cleanups are required for the hacks to actually compile and function, but the really meaningful and useful code is just what has been shown above.) All programs using framebuffer for display should be able to work now, without patching. For example, we can now run Debian installer on top of Wii-Linux by only swapping in the Wii-specific kernel with the above hacks, without touching the installer GUI module:

Debian Installer running on top of Wii-Linux Using Hacked RGB Framebuffer Driver

Debian Installer running on top of Wii-Linux Using Hacked RGB Framebuffer Driver

and from within a basic Debian, we could just ‘apt-get install’ xserver-xorg-video-fbdev and a lightweight desktop environment, e.g. LXDE (package name ‘lxde’), and we will get a full working X environment with zero patching and zero manual configuration:

Gnome Display Manager Login on Wii

Gnome Display Manager Login on Wii

LXDE desktop on Wii

LXDE desktop on Wii

IceWeasel on Wii

IceWeasel on Wii

This entry was posted in Linux, Wii and tagged , , , , , . Bookmark the permalink.

63 Responses to Hacking Up an RGB Framebuffer Driver for Wii-Linux

  1. fishears says:

    Me again :)
    Do you happen to have a copy of the BFS patch you used? Con Kolivas isn’t hosting anything pre 3.0 now. I’ve looked for one myself but they all seem to rely on different source files than the ones I have (and the ones on kernel.org – which is weird).
    Thanks if have. No worries if you don’t.

  2. fishears says:

    Not sure if you still read these but I’ve just got back into Wii Linux for writing fiction on. I’ve got your 480p kernel running with fbdev and its fantastic having no problems with colour and also having the screen fit my TV. I got b43 loading (add to /etc/modules) and now all that’s missing is swap on RVL-MEM2. I was using it just fine on the previous mikep5 2.6.32 kernel but now its gone. I’ve set up a swap file for now (which is ok but slow). Can you tell me if MEM2 is available as a device in your kernel and if so, how to get at it (it doesn’t list in /proc/devices)?

  3. DeltaLink says:

    Which driver would be best for console (TTY) use with a framebuffer (with no X) in your opinion? I’m looking for a driver with YUV colour that requires the least CPU for use with a resource heavy GUI console application. Do any of the drivers allow for resolution or bit depth reduction by default or with little modification?

    • DeltaLink says:

      I mean YUY2 colour space not YUV.

    • farter says:

      Hi. Both drivers are actually quite similar, the real difference being that the unmodified wii linux driver only hardcodes color conversion for text console, while the modified RGB driver converts everything. Neither supports bit depth or resolution change, but you can choose to use centered small area for display.

      The fastest approache IMO is to make your app directly output YUY2 images onto kernel framebuffer. If you need resolution or bit depth reduction, do it in your app too. This is how GeeXboX for Wii output images.

      Alternatively, you could use devkitppc to develop Wii mode app. I’m not sure, but devkit apps seem to be able to harness some GPU capabilities.

  4. -DarkAceZ- says:

    OK, thanks! I had already extracted the modules.zip, but forgot to run depmod.
    Is there anyway to make it automaticly do modprobe b43 on bot? I noticed that it trys to start an internet connection, but because the hardware isn’t started it can’t do anything.