File streams are now a very common attack surface, here is a high level introduction that should make you understand the design of known attacks beyond the code reading for a particular function. I already talked about FSOP here. This article reviews glibc 2.36. Most of this article comes from this awesome series of articles about the _IO_FILE strcuture.
Global design
As said in my previous writeup:
Basically on linux “everything is a file” from the character device the any stream (error, input, output, opened file) we can interact with a resource by just by opening it and getting a file descriptor on it, right ? This way each file descripor has an associated structure called FILE you may have used if you have already done some stuff with files on linux.
/* The tag name of this struct is _IO_FILE to preserve historic C++ mangled names for functions taking FILE* arguments. That name should not be used in new code. */ struct _IO_FILE { int _flags; /* High-order word is _IO_MAGIC; rest is flags. */
/* The following pointers correspond to the C++ streambuf protocol. */ char *_IO_read_ptr; /* Current read pointer */ char *_IO_read_end; /* End of get area. */ char *_IO_read_base; /* Start of putback+get area. */ char *_IO_write_base; /* Start of put area. */ char *_IO_write_ptr; /* Current put pointer. */ char *_IO_write_end; /* End of put area. */ char *_IO_buf_base; /* Start of reserve area. */ char *_IO_buf_end; /* End of reserve area. */
/* The following fields are used to support backing up and undo. */ char *_IO_save_base; /* Pointer to start of non-current get area. */ char *_IO_backup_base; /* Pointer to first valid character of backup area */ char *_IO_save_end; /* Pointer to end of non-current get area. */
struct _IO_marker *_markers;
struct _IO_FILE *_chain;
int _fileno; int _flags2; __off_t _old_offset; /* This used to be _offset but it's too small. */
/* 1+column number of pbase(); 0 is unknown. */ unsignedshort _cur_column; signedchar _vtable_offset; char _shortbuf[1];
_IO_lock_t *_lock; #ifdef _IO_USE_OLD_IO_FILE };
struct _IO_FILE_complete { struct _IO_FILE _file; #endif __off64_t _offset; /* Wide character stream stuff. */ struct _IO_codecvt *_codecvt; struct _IO_wide_data *_wide_data; struct _IO_FILE *_freeres_list; void *_freeres_buf; size_t __pad5; int _mode; /* Make sure we don't get into trouble again. */ char _unused2[15 * sizeof (int) - 4 * sizeof (void *) - sizeof (size_t)]; };
Before starting to describe each field of the structure, you have to understand that according to behaviour of the function that uses a file stream, only a small part of the _IO_FILE structure is used. For example if the file stream is byte oriented, _IO_wide_data related operations are not used.
Let’s review the fields of the structure:
_flags: High-order word is _IO_MAGIC, rest is flags.
_IO_read_ptr address of input within the input buffer that has been already used.
_IO_read_end end address of the input buffer.
_IO_read_base base address of the input buffer.
_IO_write_base base address of the ouput buffer.
_IO_write_ptr points to the character that hasn’t been printed yet.
_IO_write_end end address of the output buffer.
_IO_buf_base base address for both input and output buffer.
_IO_buf_end end address for both input and output buffer.
_chain stands for the single linked list that links of all file streams.
_fileno stands for the file descriptor associated to the file.
_vtable_offset stands for the offset of the vtable we have to use.
_offset stands for the current offset within the file.
Common functions
_IO_setb (FILE *f, char *base, char *end, int do_user_buf): Initializes the base buffer. Here is its implementation:
fp->_wide_data->_wide_vtable = jmp; } else /* Cause predictable crash when a wide function is called on a byte stream. */ fp->_wide_data = (struct _IO_wide_data *) -1L; fp->_freeres_list = NULL; }
void _IO_new_file_init_internal (struct _IO_FILE_plus *fp) { /* POSIX.1 allows another file handle to be used to change the position of our file descriptor. Hence we actually don't know the actual position before we do the first fseek (and until a following fflush). */ fp->file._offset = _IO_pos_BAD; fp->file._flags |= CLOSED_FILEBUF_FLAGS;
_IO_link_in (fp); fp->file._fileno = -1; }
fp->file._flags is initialized to CLOSED_FILEBUF_FLAGS which means according to its definition:
// https://elixir.bootlin.com/glibc/glibc-2.36/source/libio/libio.h#L78 #define _IO_TIED_PUT_GET 0x0400 /* Put and get pointer move in unison. */
Then the fp (the file pointer) is linked into the single linked list that keeps track of every file stream, for which the HEAD is _IO_list_all. _IO_link_in is defined like this:
if (__wcsmbs_named_conv (&fcts, ccs[2] == '\0' ? upstr (ccs, cs + 5) : ccs) != 0) { /* Something went wrong, we cannot load the conversion modules. This means we cannot proceed since the user explicitly asked for these. */ (void) _IO_file_close_it (fp); free (ccs); __set_errno (EINVAL); returnNULL; }
/* Clear the state. We start all over again. */ memset (&fp->_wide_data->_IO_state, '\0', sizeof (__mbstate_t)); memset (&fp->_wide_data->_IO_last_state, '\0', sizeof (__mbstate_t));
That’s a pretty huge function but most of it is just parsing and handling of a specific encoding for the file. First it checks if the file is already open then it parses the mode and once it’s done it calls _IO_file_open with the right flags. Then is the file requires a specific encoding it intitializes _wide_data and so on to properly handle it. Let’s take a look at the _IO_file_open function:
FILE * _IO_file_open (FILE *fp, constchar *filename, int posix_mode, int prot, int read_write, int is32not64) { int fdesc; if (__glibc_unlikely (fp->_flags2 & _IO_FLAGS2_NOTCANCEL)) fdesc = __open_nocancel (filename, posix_mode | (is32not64 ? 0 : O_LARGEFILE), prot); else fdesc = __open (filename, posix_mode | (is32not64 ? 0 : O_LARGEFILE), prot); if (fdesc < 0) returnNULL; fp->_fileno = fdesc; _IO_mask_flags (fp, read_write,_IO_NO_READS+_IO_NO_WRITES+_IO_IS_APPENDING); /* For append mode, send the file offset to the end of the file. Don't update the offset cache though, since the file handle is not active. */ if ((read_write & (_IO_IS_APPENDING | _IO_NO_READS)) == (_IO_IS_APPENDING | _IO_NO_READS)) { off64_t new_pos = _IO_SYSSEEK (fp, 0, _IO_seek_end); if (new_pos == _IO_pos_BAD && errno != ESPIPE) { __close_nocancel (fdesc); returnNULL; } } _IO_link_in ((struct _IO_FILE_plus *) fp); return fp; } libc_hidden_def (_IO_file_open)
If the mode doesn’t allow the open process to be a cancellation point it calls __open_nocancel, else it calls __open. When the file is open, it initializes flags, file descriptor (fileno) and links the actual file pointer to the single linked list that stores every file stream (if that’s not already the case).
Then we’re back into __fopen_internal to call __fopen_maybe_mmap on the newly open file:
FILE * __fopen_maybe_mmap (FILE *fp) { #if _G_HAVE_MMAP if ((fp->_flags2 & _IO_FLAGS2_MMAP) && (fp->_flags & _IO_NO_WRITES)) { /* Since this is read-only, we might be able to mmap the contents directly. We delay the decision until the first read attempt by giving it a jump table containing functions that choose mmap or vanilla file operations and reset the jump table accordingly. */
I think the comment is enough explicit, once __fopen_maybe_mmap is called the fp is returned given the file descriptor has properly been allocated, initialized and that the file is open. Else it means that there are some errors, then the fp is unlinked from the single linked that stores every file stream, and the locked_FILE is freed, returning NULL indicating an error.
That’s basically how fopen works !
fread
Once a _IO_FILE structure has been initialized and linked into the _IO_list_all single linked list, several operations can occur. A basic primitive would be to read data from a file, that’s what fread does with the use of certain fields of _IO_FILE.
According to the man: “The function fread() reads nmemb items of data, each size bytes long, from the stream pointed to by stream, storing them at the location given by ptr.”. Now let’s dig deeper within the code. fread is defined there:
If the amount of requested bytes is null, zero is returned. CHECK_FILE checks (if IO_DEBUG is enabled) if fp exists and if fp->flags is properely structured:
When this is the first time _IO_sgetn is called, on most of the platforms (the one which support mmap) the vtable is initialized to _IO_file_jumps_maybe_mmap:
staticsize_t _IO_file_xsgetn_maybe_mmap (FILE *fp, void *data, size_t n) { /* We only get here if this is the first attempt to read something. Decide which operations to use and then punt to the chosen one. */
decide_maybe_mmap is basicaly trying to map the file, if it succeeds the vtable is initialized to &_IO_file_jumps_mmap else it’s initialized to &_IO_file_jumps. The function is pretty easy to read, except maybe for the S_ISREG (st.st_mode) && st.st_size != 0 that checks if it is a regular file and if its size isn’t null. Here is the full code:
staticvoid decide_maybe_mmap(FILE *fp) { /* We use the file in read-only mode. This could mean we can mmap the file and use it without any copying. But not all file descriptors are for mmap-able objects and on 32-bit machines we don't want to map files which are too large since this would require too much virtual memory. */ struct __stat64_t64st;
if (_IO_SYSSTAT (fp, &st) == 0 && S_ISREG (st.st_mode) && st.st_size != 0 /* Limit the file size to 1MB for 32-bit machines. */ && (sizeof (ptrdiff_t) > 4 || st.st_size < 1*1024*1024) /* Sanity check. */ && (fp->_offset == _IO_pos_BAD || fp->_offset <= st.st_size)) { /* Try to map the file. */ void *p;
p = __mmap64 (NULL, st.st_size, PROT_READ, MAP_SHARED, fp->_fileno, 0); if (p != MAP_FAILED) { /* OK, we managed to map the file. Set the buffer up and use a special jump table with simplified underflow functions which never tries to read anything from the file. */
Two operations are very important to note in this function. First, _IO_setb (take a look at this) is called to initialize the begin of the base buffer to the begin of the memory mapping of the file, the end of the base buffer is then initialized to the end of the file (p + st.st_size). Right after _IO_setg (take a look at this) is called to initialize the read buffer of the file, the base of the read buffer is initialized to the mapping of the file, the current pointer to p + fp->_offset and the end of the buffer to the end of the file mapping.
Then according to what vtable is used, the xsgetn is distinct:
if (have < n) { if (__glibc_unlikely (_IO_in_backup (fp))) { s = __mempcpy (s, read_ptr, have); n -= have; _IO_switch_to_main_get_area (fp); read_ptr = fp->_IO_read_ptr; have = fp->_IO_read_end - fp->_IO_read_ptr; }
if (have < n) { /* Check that we are mapping all of the file, in case it grew. */ if (__glibc_unlikely (mmap_remap_check (fp))) /* We punted mmap, so complete with the vanilla code. */ return s - (char *) data + _IO_XSGETN (fp, data, n);
read_ptr = fp->_IO_read_ptr; have = fp->_IO_read_end - read_ptr; } }
if (have < n) fp->_flags |= _IO_EOF_SEEN;
if (have != 0) { have = MIN (have, n); s = __mempcpy (s, read_ptr, have); fp->_IO_read_ptr = read_ptr + have; }
/* Guts of underflow callback if we mmap the file. This stats the file and updates the stream state to match. In the normal case we return zero. If the file is no longer eligible for mmap, its jump tables are reset to the vanilla ones and we return nonzero. */ staticint mmap_remap_check(FILE *fp) { structstat64st;
if (_IO_SYSSTAT (fp, &st) == 0 && S_ISREG (st.st_mode) && st.st_size != 0 /* Limit the file size to 1MB for 32-bit machines. */ && (sizeof (ptrdiff_t) > 4 || st.st_size < 1*1024*1024)) { constsize_t pagesize = __getpagesize (); # define ROUNDED(x) (((x) + pagesize - 1) & ~(pagesize - 1)) if (ROUNDED (st.st_size) < ROUNDED (fp->_IO_buf_end - fp->_IO_buf_base)) { /* We can trim off some pages past the end of the file. */ (void) __munmap (fp->_IO_buf_base + ROUNDED (st.st_size), ROUNDED (fp->_IO_buf_end - fp->_IO_buf_base) - ROUNDED (st.st_size)); fp->_IO_buf_end = fp->_IO_buf_base + st.st_size; } elseif (ROUNDED (st.st_size) > ROUNDED (fp->_IO_buf_end - fp->_IO_buf_base)) { /* The file added some pages. We need to remap it. */ void *p; #if _G_HAVE_MREMAP p = __mremap (fp->_IO_buf_base, ROUNDED (fp->_IO_buf_end - fp->_IO_buf_base), ROUNDED (st.st_size), MREMAP_MAYMOVE); if (p == MAP_FAILED) { (void) __munmap (fp->_IO_buf_base, fp->_IO_buf_end - fp->_IO_buf_base); goto punt; } #else (void) __munmap (fp->_IO_buf_base, fp->_IO_buf_end - fp->_IO_buf_base); p = __mmap64 (NULL, st.st_size, PROT_READ, MAP_SHARED, fp->_fileno, 0); if (p == MAP_FAILED) goto punt; #endif fp->_IO_buf_base = p; fp->_IO_buf_end = fp->_IO_buf_base + st.st_size; } else { /* The number of pages didn't change. */ fp->_IO_buf_end = fp->_IO_buf_base + st.st_size; } # undef ROUNDED
/* If we are already positioned at or past the end of the file, don't change the current offset. If not, seek past what we have mapped, mimicking the position left by a normal underflow reading into its buffer until EOF. */
return0; } else { /* Life is no longer good for mmap. Punt it. */ (void) __munmap (fp->_IO_buf_base, fp->_IO_buf_end - fp->_IO_buf_base); punt: fp->_IO_buf_base = fp->_IO_buf_end = NULL; _IO_setg (fp, NULL, NULL, NULL); if (fp->_mode <= 0) _IO_JUMPS_FILE_plus (fp) = &_IO_file_jumps; else _IO_JUMPS_FILE_plus (fp) = &_IO_wfile_jumps; fp->_wide_data->_wide_vtable = &_IO_wfile_jumps;
return1; } }
x First is computed the amount of bytes that contains the read buffer (have). If we do not have the right amount of bytes within the read buffer we first try to copy data from the read buffer. Then we check we are mapping the whole file (and not only a part of it) with the use of mmap_remap_check (to avoid useless code I put it directly after the implementation of _IO_file_xsgetn_mmap), if it fails the file is unmapped and the vanilla file operations is used to read data from the file.
