RAM FileSystem and expand

The RAM FileSystem

---

uClinux has a RAM file system (ramdisk) for its scratch pad /tmp and /var areas in the absence of a hard disk drive. The current distribution of uClinux痴 romdisk comes with a ramfs of 256kbytes unformatted and is missing a /var/tmp directory causing a cd tmp to report a bad directory. The /tmp directory in the root is a symbolic link which points to /var/tmp. var is the mount point for the RAM file system.

The ramfs is uncompressed and mounted in the rc startup script,

 # expand the ramdisk
 /sbin/expand /ramfs.img /dev/ram0

 # mount ramdisk, proc and nfs
 /bin/mount -t ext2 /dev/ram0 /var

It consists of a ext2 filesystem which is compressed using a Zero Run Length Encoding algorithm (ZRTE). This is uncompressed using the expand utility to the RAM block device /dev/ram0. Once this process has been completed it is then mounted as an ext2 file system with the mount point of /var.

If we have a need to fix the tmp link, we could simply add a command in the rc startup script to make a tmp directory in var after the ramfs has been mounted. However if you need to increase the size of the ramdisk, this isn稚 as simple. You will need to create a new ramfs.img. In some applications a larger ramdisk is required. You may want to log data to a file that is then downloaded using anonymous FTP, HTTP etc.

Creating a new ramfs

---

Creating a new ramfs is a reasonably simple task. One of the problems you need to watch is byte order, whether your system is big endian or little endian.

Start by zeroing out the ram block device. Later, we will use a zero run length compression utility, thus in the interests of getting a more compressible image, it痴 recommended you carry out this step. You will also need to choose what size a ramdisk you need. A larger ramdisk will allow you to store more logs or temporary files but at the expense of system RAM. It will normally depend upon what applications you are using with your uClinux system.

 # dd if=/dev/zero of=/dev/ram0 bs=1k count=1024
 1024+0 records in
 1024+0 records out    

Next we make a second extended file system. The 没 flag turns on verbose mode so we can see some of the stats. By default 5% of the blocks is reserved for superuser. We turn this off using 卜0. We also turn off any extra features of the ext2fs using -Onone. On post 2.2 kernel systems, sparse_super and filetype options are automatically turned on when creating an ext2fs. However both these features are not correctly supported by kernels pre 2.2 and as a result mount will report a failure when mounting the filesystem on your 2.0.38 uClinux system.

 # mke2fs -vm0 -Onone /dev/ram0 1024
 mke2fs 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09
 Filesystem label=
 OS type: Linux
 Block size=1024 (log=0)
 Fragment size=1024 (log=0)
 128 inodes, 1024 blocks
 0 blocks (0.00%) reserved for the super user
 First data block=1
 1 block group
 8192 blocks per group, 8192 fragments per group
 128 inodes per group
 
 Writing inode tables: done
 Writing superblocks and filesystem accounting information: done   

The next step is to add extra folders or files to the ramdisk. This is an optional procedure thus you can omit this step, if not needed. By default when the ext2 filesystem is created a lost+found directory is added. This will be present in /var/lost+found once mounted on your uClinux system. It is also recommended to create a tmp folder so the tmp symbolic link works. To add files and folders, first mount the file system as ext2.

 # mount -t ext2 /dev/ram0 /mnt/ramdisk 
 # cd /mnt/ramdisk
 # mkdir tmp

Then once you are finished, unmount the file system to ensure a clean mount in the future.

 # umount /mnt/ramdisk

Now it痴 time to move the filesystem from the ram block device to a local file. This copies the file system byte for byte to ramfsraw.img. For a 1MB ramfs, this file will be 1MB in size.

 # dd if=/dev/ram of=ramfsraw.img bs=1k count=1024
 1024+0 records in
 1024+0 records out   

It would be quite inefficient to add this 1Mbyte file to the romdisk in its current form, and since the majority of the filesystem is free, we can compress it extremely efficiently using ZRLE (Zero Run Length Encoding). You may see some documentation describing the process as 杜aking a file with holes・

What ZRLE does is record the blocks of non-zero data, prefixing it with the length of the block and its position in the file. As most of the data is zero, it can strip this out. The expand utility will simply zero out the entire length of the expanded file, then copy the blocks of data back to its relevant locations indicated by its position header.

The compression can be done with a utility called holes. There are a couple of versions floating around which have endian and/or block size reporting problems. You can download the source and compiled binaries from These binaries will run on Linux little endian platforms.

 # /holes/holes ramfsraw.img >ramfs.img
 Length of original file 1048576

Listing the two files will show the run length compression at work. Now we can add it to our romdisk and not waste considerable space. Of course if you do put some files in the ramdisk, then it won稚 compress as well.

 # ls ram* -l
 -rw-r--r--     1 root      root          2639 Mar   4 16:00 ramfs.img
 -rw-r--r--     1 root      root       1048576 Mar   4 15:59 ramfsraw.img 

You may also notice that our generated ramfs.img is smaller than the uClinux stock ramfs.img (3340), even that an extra folder has been added. Just another reason why you should update and generate your own. (For some reason, the run length block sizes on the stock ramfs are no larger than 200-300 bytes even though expand.c allocates a 2048 byte buffer when expanding the image).

Testing your RAM filesystem

You can test your newly created ramfs on your uClinux platform by firstly unmounting the existing ramfs and then expanding and mounting your new ramfs.

 /bin/umount /var
 /sbin/expand /ramfs.img /dev/ram0
 /bin/mount -t ext2 /dev/ram0 /var

However you may also wish to test it on your linux desktop. Coping the expand.c file from /src/init/ and recompiling it for x86 linux is going to lead you into endian problems. Either fix up the ntohl() define, or use the precompiled expand binary for x86 from the tarball.


======================================================================

/* expand.c: expand a file with holes into another
 *
 * Copyright (C) 1998  Kenneth Albanowski ,
 *                     D. Jeff Dionne ,
 *                     The Silver Hammer Group, Ltd.
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 */
 
#include
#include
#include
#include
#include
#include

//#define ntohl(x) (x)
#include

int
main(int argc, char *argv[])
{
  int fdi;
  int fdo;
  int block = 0;
  unsigned int pos;
  unsigned int len;
  unsigned int count;
  char *buf;

  count = 0;

  if (argc < 3) {
    fprintf(stderr,"usage: %s fromfile tofile\n");
    exit(1);
  }

  if ((fdi = open(argv[1],O_RDONLY)) < 0) {
    fprintf(stderr,"Can't open compressed file %s\n",argv[1]);
    exit(2);
  }

  if ((fdo = open(argv[2],O_WRONLY | O_CREAT | O_TRUNC, 0666)) < 0) {
    fprintf(stderr,"Can't open expanded file %s\n",argv[2]);
    close(fdi);
    exit(3);
  }

  if (!(buf = malloc(2048))) {
    fprintf(stderr,"can't allocate memory\n");
    close(fdi);
    close(fdo);
    exit(10);
  }

  /* Prefill */
  read(fdi,&len,4);
  len = ntohl(len);
  printf("Expanded file (%s) length is %d\nFilling file with zeros . . ",argv[2],len);

  memset(buf, 0, 2048);
  while(len>0) {
      write(fdo, buf, (len > 2048) ? 2048 : len);
      len -= 2048;
  }
  lseek(fdo, 0, SEEK_SET);
  printf("Done.\nNow start with ZRLE :\n");


  /* ZRLE */
  while (read(fdi,&pos,4) == 4) {
    if (read(fdi,&len,4) != 4) break;
    pos = ntohl(pos);
    len = ntohl(len);
    printf(" %dbyte block at 0x%04X\n",len,pos);   
    block++;
    lseek(fdo,pos,SEEK_SET);
    read(fdi,buf,len);
    write(fdo,buf,len);
  }

  printf("Finished processing %d blocks.\n",block);
  close(fdi);
  close(fdo);
  free(buf);

  exit(0);
}

===================================================

/* make a file with holes */

#include
#include
#include
#include
#include
#include

char buf[2048];

int
main(int argc, char *argv[])
{
  int block=0;
  int count=0;
  int pos=0;
  int zcount=0;
  int b;
  int len;
  int in = open(argv[1], O_RDONLY);

  char ch = 0;

  // A second argument can be specified dictating the default char to ignore.
  // If this is not present assume ZERO. (ZRLE)
  if (argc > 2)
    {
      ch = atoi(argv[2]);
      fprintf(stderr,"Specified 2nd Argument %c %02X\n",ch,ch);
    } 

  // Find End of File. Report Size.
  len = lseek(in, 0, SEEK_END);
  fprintf(stderr,"Length of original file %d\n",len);
  len = htonl(len);
  write(1, &len, 4); // Write Length of File

  // Reset to Start of File.
  lseek(in, 0, SEEK_SET);


  while(read(in, &buf[count], 1) > 0)
   {
    pos++; // Current Position

    if (count || (buf[count] != ch)) {
      if (buf[count] == ch)   // Vadim
        zcount++;
      count ++;

    // Expand has a maximum buffer size of 2048 bytes, we want to
    // keep blocks >= 8 bytes for effiency.
    if ((zcount == 8) || (count == 2048))
          {
       fprintf(stderr," Writing Block %d Start Address %04X Length %d\n",block++,pos-count,count-1);

       // Write Starting Position
       b = htonl(pos-count);
       write(1,&b,4);

       // Write Length of Block
           b = htonl(count-1);
          write(1,&b,4);

       // Write Block
            write(1,buf,count-1);
      
           // Reset counters
           count = zcount = 0;
      }
    }
  }

  if (count) {
    b = htonl(pos-count);
    write(1,&b,4);
    b = htonl(count);
    write(1,&b,4);
    write(1,buf,count);
  }
}