Need to integrate a custom filesystem into the ESP32 framework. This is what I learned from doing just that.

Introduction

The ESP32 is a small, but capable, embedded CPU. It comes in a few flavours but generally it carries some RAM, WiFi and Bluetooth stacks. It has 3 HW UARTS and a bunch of other GPIO pins. It can be programmed either through the Arudino environment or directly with the Espressif IDF (IoT Development Framework). While the Arduino allows you to build almost anything the Espressif IDF is much more powerful. If you need it or not - up to you. This guide can be used regardless. As the Arduino framework is built on-top of the Espressif IDF you can hook in directly to it through the Arduino framework.

VFS

The VFS is a virtual file system layered on top of the actual filesystem. It allows you to mount various filesystems in a single structure in an apparent seemingless way. The VFS will look at the path you are accessing and redirect the operation to the appropriate underlying filsystem driver.

Even if the integration is pretty straight forward it is pooly documented.

The procedure is:

• Delcare you VFS
• Mount and Register it to the ESP-IDF

Once the above has been done, any regular file access methods (fopen/fclose/fread/fwrite/etc…) for a file under your mountpoint will call down to your functions.

VFS Registration

The VFS functions will act as a proxy to the filesystem you are integrating. The documentation states you can leave any unimplemented function in the VFS interface to NULL. While this is true the documentation doesnt state the bare minimum implementation in order for it to work. You’ll need:

• Open
• Close
• Read
• Write
• Fstat

Details on actual function prototypes can be found under: esp-idf/components/vfs/include/esp_vfs.h

This is what I think is a bare minium file system declaration.

static ssize_t vfs_write(void *ctx, int fd, const void * data, size_t size);
static ssize_t vfs_read(void *ctx, int fd, void *dst, size_t size);
static int vfs_open(void *ctx, const char *path, int flags, int mode);
static int vfs_close(void *ctx, int fd);
static int vfs_fstat(void *ctx, int fd, struct stat *st);
static off_t vfs_lseek(void* ctx, int fd, off_t offset, int mode);

static esp_vfs_t myfs = {
    ESP_VFS_FLAG_CONTEXT_PTR,   // Leave this to ESP_VFS_FLAG_DEFAULT if you don't need the context
    &vfs_write,
    &vfs_lseek, // Not sure this is actually needed
    &vfs_read,
    &vfs_open,
    &vfs_close,
    &vfs_fstat,
};

static void MountMyFS() {
    // Omitting stuff here
    esp_vfs_register("/data",&myfs,your_context_ptr);
}

NOTE: Regarding the context pointer, if not needed, you’ll need to remove the ctx argument in the function prototype.

For more details: https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/storage/vfs.html

Once registered any access to data under /data will be redirected to your functions.

Examples:

static void TestMyFS() {
    FILE *f = fopen("/data/myfile.txt","r");
    fclose(f);
}

The fopen call will look at the full path and deduct that /data belongs the your VFS. Anything after the mountpoint will be sent down to you, in this case vfs_open will be called with the path argument set to /myfile.txt.

The first time around I would suggest you create a very bare bone implementation that just outputs the call identifier on the serial. This will allow you to see how the API behaves during read/write and give you an idea of the calls and parameters involved.

Example on an empty VFS:

static int vfs_open(void *ctx, const char *path, int flags, int mode) {
    printf("vfs_open, called - not implemented\n");
    // Do return something here - otherwise the rest will never be called
    return 10;
}
static int vfs_close(void *ctx, int fd) {
    printf("vfs_fstat, called - not implemented\n");
    return 0;
}
static ssize_t vfs_read(void *ctx, int fd, void *dst, size_t size) {
    printf("vfs_read, called - not implemented\n");
    return size;
}
static ssize_t vfs_write(void *ctx, int fd, const void * data, size_t size) {
    printf("vfs_write, called - not implemented\n");
    return size;
}
static int vfs_fstat(void *ctx, int fs, struct stat *st) {
    printf("vfs_fstat, called - not implemented\n");
    return 0;
}
static off_t vfs_lseek(void* ctx, int fd, off_t offset, int mode) {
    printf("vfs_lseek, called - not implemented\n");
    return 0;
}

VFS Implementation

This portion is up to you. It can essentially be anything supporting open/read/write/close. However, here are my notes for the various functions

COMMENT: I have omitted several things from this code - specically regarding error handling. When something goes wrong you should set the approriate errno value.

Open

Prototype with context:

static int vfs_open(void *ctx, const char *path, int flags, int mode);

The path argument is everything after your mountpoint - including any slash or ’.’ values (see above).

The mode represents a bit field for the regular fopen flags. They are converted on the fly somewhere in the API chain and you need to either reuse the API flags or convert them to your own types. Problem is that the type and the definition is not declared anywhere in the documentation. You find them under: esp-idf/components/newlib/include/sys/_default_fcntl.h However, don’t include that one - better to include the standard file esp-idf/components/newlib/include/sys/fcntl.h

The return value is a file-descriptor. This value is your to choose but it must reside within 0..FD_SETSIZE-1. The default FD_SETSIZE is ‘64’ and you should not return values above this. I tried to go outside of this range as it stated that the FD is local to your driver but I kept on getting very strange FD values in subsequent calls.

I found code for the mode-conversion in the SPIFFS implementation for ESP32. I modified it to my purpose.

static int vfs_open(void *ctx, const char *path, int flags, int mode) {
    // Without the context your filesystem is a singleton
    MyFileSystem *pfs = (MyFileSystem *)ctx;

    int pfsMode = vfs_mode_conv(mode);
    pfsFile = fs_openfile(pfs, path, pfsMode);
    if (pfsFile == NULL) {
        printf("vfs_open, ERR: unable to open file, pfserr: %s\n", fs_error(pfs));
        return -1;
    } 
    prrintf("vfs_open, ok file: %s, mode: %.2x (%.2x)\n", path, pfsMode,mode);
    printf("  FD_SETSIZE: %d\n", FD_SETSIZE);        
    // Return 0..FD_SETSIZE-1 here
    return fd_from_file(pfs_file);
}
static int vfs_mode_conv(int m)
{
    int res = 0;
    int acc_mode = m & O_ACCMODE;
    if (acc_mode == O_RDONLY) {
        res |= MYFS_OPEN_READ;
    } else if (acc_mode == O_WRONLY) {
        res |= MYFS_OPEN_WRITE;
    } else if (acc_mode == O_RDWR) {
        res |= MYFS_OPEN_READ | MYFS_OPEN_WRITE;
    }
    if (m & O_APPEND) {
        res |= MYFS_OPEN_APPEND;
    }
    return res;
}

Close

Prototype with context:

static int vfs_close(void *ctx, int fd);

The fd is the same file descriptor you returned from open.

Close is a straight forward function. Just dispose on any resources or cached data your have related to a specific file.

static int vfs_close(void *ctx, int fd) {
    MyFileSystem *pfs = (MyFileSystem *)ctx;

    pfsFile = file_from_fd(fd); 
    
    if (pfsFile == NULL) {
        printf("vfs_close, ERR: file not opened, err: %s\n", fs_error(pfs));
        return -1;
    }
    if (fs_closefile(pfs, pfsFile) != FS_OK) {
        printf("vfs_close, ERR: can't close file, err: %s\n", fs_error(pfs));
        return -1;
    }
    printf("vfs_close, ok file closed\n");
    free_file(pfsFile);
    return 0;
}

Read

Prototype with context:

static ssize_t vfs_read(void *ctx, int fd, void *dst, size_t size);

The ```fd“ is your regular file-descriptor.

The dst is a buffer from the framework - it is NOT the buffer you supplied down to fread.

The size is the number of bytes requested.

You might remember that the prototype for fread looks like:

 size_t fread(void *restrict ptr, size_t size, size_t nitems, FILE *restrict stream);

Where size is the size of ONE element and the nitems specifies the number of elements to read. The return value is actually not bytes but rather the number of elements read. As you can see, at the VFS level, we simply don’t know these values.

The expected return value from your VFS implementation is not documented. However it should be the number of bytes read or a negative value if an error occured. Zero indicates EOF.

In the case of EOF you should expect multiple call’s down to the VFS. The ESP-IDF framework will try to read beyond EOF and you should in that case return 0 as an indicator that you have reached the end of file and that there is no more data.

Example (assuming the file only has four bytes):

void ReadFile() {

    FILE *f = fopen("/data/myfile.txt","r");
    char buffer[256];
    size_t res = fread(buffer,1,256,f);   
    fclose(f);
}

The ```fread“ function will try to read 256 bytes but the file only has four bytes. The following will happen:

1. The API will call to your vfs_read with request for 256 bytes
2. You will fetch 4 bytes from the file and return ‘4’
3. The API will see it did not get all 256 bytes and issue another call with request for 252 bytes
4. You should not detect an attempt to read outside the file and return ‘0’ (zero) to indicate EOF

This behaviour is not documented and I have no clue if it is common.

Example implementation:

static ssize_t vfs_read(void *ctx, int fd, void *dst, size_t size) {
    MyFileSystem *pfs = (MyFileSystem *)ctx;
    pfsFile = file_from_fd(fd); 
    if (pfsFile == NULL) {
        printf("vfs_read, ERR: file not opened\n");
        return -1;
    }
    printf("vfs_read, dst pointer is: %p\n", dst);
    ssize_t ret = fs_readfile(pfs, pfsFile, dst, size);
    printf("vfs_read, got: %d\n", ret);
    hexdump((uint8_t *)dst, 32);
    return ret;
}

Write

Prototype with context:

static ssize_t vfs_write(void *ctx, int fd, const void * data, size_t size);

The write function is straight forward. The data and the size of data is supplied in the arguyments and you should just commit this data to your implementation. You should return a the number of bytes written or a negative value if you failed.

Example implementation:

static ssize_t vfs_write(void *ctx, int fd, const void * data, size_t size) {
    MyFileSystem *pfs = (MyFileSystem *)ctx;
    pfsFile = file_from_fd(fd); 
    if (pfsFile == NULL) {
        printf("vfs_write, ERR: file not opened\n");
        return -1;
    }
    return fs_writefile(pfs, pfsFile, data, size);
}

FStat

static int vfs_fstat(void *ctx, int fd, struct stat *st);

I only use this to hint back the filesize. I am not actually sure you need to implement it. The API will issue a call down to the FSTAT function on various occasions so I found it a good idea to implement it.

I found an implementation in the SPIFFS integration - I used it and stripped what I did not need.

Example implementation

static int vfs_fstat(void *ctx, int fs, struct stat *st) {
    MyFileSystem *pfs = (MyFileSystem *)ctx;
 
    pfsFile = file_from_fd(fd); 
    int32_t sz = fs_filesize(pfs, pfsFile);

    printf("vfs_fstat, sz: %d\n", sz);

    st->st_size = sz; //s.size;
    st->st_mode = S_IRWXU | S_IRWXG | S_IRWXO | S_IFREG;    // No clue actually
    st->st_mtime = 0; 
    st->st_atime = 0;
    st->st_ctime = 0;
    return 0;
}

Implementation of the low-level driver

From the code above I use the VFS layer as just a bridge/proxy down to my actual filesystem implementation. This allows me to implement them regardless of each other. It’s a nice separation of concerns, at least for me. There is nothing stopping you from adding all filesystem logic directly to the VFS implementation. Personally I would not do that.

The simplest driver would be one that supports just a single file of a maximum size. Reading and writing would be to a memory buffer in this case. The memory buffer can statically declarated.

NOTE: I’ve never tested this example - just as a hint.

Example:

static uint8_t data[65536]; // 64k buffer
static int fs_open(MyFileSystem *filesystem, const char *path, int mode) {
    if (filesystem->isopen) {
        return -1;
    }
    filesystem->isopen = true;
    // Truncate the file on 'write' - no support for append
    if (mode & MYFS_OPEN_WRITE) {
        filesystem->filesize = 0;
    }
    filesystem->rwcursor = 0;
    return 1;
}

static int fs_close(MyFileSystem *filesystem, int myfile) {
    if ((filesystem->isopen) && (myfile == 1)) {
        filesystem->isopen = false;

        return 1;
    }
    return -1;
}

static int fs_open(MyFileSystem *filesystem, int myfile) {
    return filesystem->filesize;
}

static int fs_write(MyFileSystem *filesystem, int myfile, const void *data, size_t size) {
    if (!filesystem->isopen) {
        return -1;
    }
    // 'write' 
    memcpy(&data[filesystem->rwcursor], data, size);
    // advance the cursor
    filesystem->rwcursor += size;
    // advance the size
    filesystem->filesize += size;
    return size;
}

static int fs_read(MyFileSystem *filesystem, int myfile, void *data, size_t size) {
    if (!filesystem->isopen) {
        return -1;
    }
    size_t nBytesToCopy = size;
    if ((filesystem->rwcursor + nBytesToCopy) > filesystem->filesize) {
        nBytesToCopy = filesystem->filesize - filesystem->rwcursor;
    }
    // EOF
    if (nBytesToCopy == 0) {
        return 0;
    }
    // 'read' 
    memcpy(data, &data[filesystem->rwcursor], size);
    // advance the cursor
    filesystem->rwcursor += nBytesToCopy;
    return nBytesToCopy;
}