How do file formats really work?

So I was reading a bit about SQLite and I like its model: self-contained, truly serverless and embedded in the application. But I got curious. How does SQLite really create a database and interact with it? And this led me to a bigger question, how do programs actually work with binary files and how do file formats work?

youd probably enjoy a lot of the books in the gentoomans book torrent, i just checked it out today myself. http://g.sicp.me/books/ there's two categories 'Data Structures' and 'Databases' full of books.

Most binary file formats are very straight forward. You can look at the PNG spec and see that it starts with a special mark followed by so many bytes for this, so many bytes for that, and the header data describes the image data. You can then open some C source code that reads PNG files and follow it. It will open a file stream, seek to a position, read some bytes, rinse and repeat. Like most files, you read and write the whole PNG file in one go and then you're done. SQLite by contrast dances on its database file and the file structure is optimized for that. I don't understand all the features and details but you can read about it here: https://www.sqlite.org/fileformat2.html

In a very general sense, binary files typically begin with a header that contains some series of bytes that identifies the file as part of the format, then some info about the file, like a byte for the version of the format and flags for options. Some binary formats include the size of the file and checksums of the data within them so that file corruption can be detected. Most of the time, there is a specification that describes the layout, although sometimes you'll find projects where the authors just rely on their language's binary serialization libraries to write some data structure to a file and call it a day. You can look up the specification for any non-proprietary file format, and many proprietary ones, to learn about them.

SQL databases usually have an underlying block model for interacting with the disk. So your file format is really just a bunch of blocks, not unlike how a file system works, and the SQL abstractions are built on top of that. For specifics of SQLite or any other program you will need to look at their documentation or code to really know how the binary file format works. Usually there are many layers of abstraction because working directly with serial data is not very pleasant.

A bit late, but I figured this thread needed a detailed answer. This one assumes you know the basics of C. Here's how reading a binary file typically works: >first few bytes are a magic number which identifies the format and may also be used to infer which endianness the file was written with (this depends on the format) >then you have metadata you read with calls to the fread() C function, potentially adjusting for endianness >some fields contains offsets that effectively act as pointers: use fseek() to go where they point to, and find a data structure to read with fread() calls there To write a binary file, you usually do this: >call to fseek() to skip the first bytes and leave enough room for the magic number and metadata >write data using fwrite() >use ftell() to know your position and use it to write offset pointers (or keep track of your position in a variable if you can) >assuming the data structure in your file is like a tree, you will tend to write the upper nodes later, it doesn't really matter since reading the file will involve moving all over the place with fseek() >when everything is done, fseek() to the beginning and write the metadata, including the offsets to the data itself (writing the data before the metadata allows you to write the metadata in one pass with all the offsets already computed) Additional notes: >whether fseek() to go to the end of a binary file is well defined or UB depends on the OS (on POSIX, it works) >therefore, the only portable way to measure the size of a binary file is to count how many times you can call fgetc() before the eof flag is raised (which may never happen if you open a device file like /dev/zero) >there may be unused bytes in binary files, but they usually don't matter >C integer types like short, long, etc have implementation-defined width, but in binary files all widths must be fixed, so the stdint.h header is mandatory >on Windows, long is 32-bit, meaning fseek() and ftell() can't be used on files over 4 GB; the Windows CRT has other functions that use long long instead of long >Windows has popularized the practice of calling fread() and fwrite() on entire structures, but C allows padding bytes to be inserted between structure members, so it's more portable to do one call per structure members (though arrays are guaranteed to have no padding) >for databases, you want to make it so that if the machine crashes anytime but after the very last byte written, the database ends up in the state it was before the whole operation instead of some intermediary state; explaining how to ensure this requires entire books, but a simple approach is to have metadata with two offset fields and an extra byte to tell which offset field is in use, write data by making copies of all blocks that are modified, and finalizing the changes by flipping that extra byte

>>32741 thank you for writing this >UB ugly bastard >CRT C runtime

>>32741 for me though? it's mmap

I recommend you look at kaitaistruct to investigate how binary formats work https://kaitai.io/ https://formats.kaitai.io/sqlite3/

What is that tool in picrel? Is it a steganography tool?

>>37464 No it's just hexyl, a tool to print a file's data in hex: https://github.com/sharkdp/hexyl

>>32543 Not OP but this look amazing thanks a lot.

>>32741 >A bit late, but I figured this thread needed a detailed answer. This one assumes you know the basics of C. > >Here's how reading a binary file typically works: >first few bytes are a magic number which identifies the format and may also be used to infer which endianness the file was written with (this depends on the format) >then you have metadata you read with calls to the fread() C function, potentially adjusting for endianness >some fields contains offsets that effectively act as pointers: use fseek() to go where they point to, and find a data structure to read with fread() calls there > >To write a binary file, you usually do this: >call to fseek() to skip the first bytes and leave enough room for the magic number and metadata >write data using fwrite() >use ftell() to know your position and use it to write offset pointers (or keep track of your position in a variable if you can) >assuming the data structure in your file is like a tree, you will tend to write the upper nodes later, it doesn't really matter since reading the file will involve moving all over the place with fseek() >when everything is done, fseek() to the beginning and write the metadata, including the offsets to the data itself (writing the data before the metadata allows you to write the metadata in one pass with all the offsets already computed) > >Additional notes: >whether fseek() to go to the end of a binary file is well defined or UB depends on the OS (on POSIX, it works) >therefore, the only portable way to measure the size of a binary file is to count how many times you can call fgetc() before the eof flag is raised (which may never happen if you open a device file like /dev/zero) >there may be unused bytes in binary files, but they usually don't matter >C integer types like short, long, etc have implementation-defined width, but in binary files all widths must be fixed, so the stdint.h header is mandatory >on Windows, long is 32-bit, meaning fseek() and ftell() can't be used on files over 4 GB; the Windows CRT has other functions that use long long instead of long >Windows has popularized the practice of calling fread() and fwrite() on entire structures, but C allows padding bytes to be inserted between structure members, so it's more portable to do one call per structure members (though arrays are guaranteed to have no padding) >for databases, you want to make it so that if the machine crashes anytime but after the very last byte written, the database ends up in the state it was before the whole operation instead of some intermediary state; explaining how to ensure this requires entire books, but a simple approach is to have metadata with two offset fields and an extra byte to tell which offset field is in use, write data by making copies of all blocks that are modified, and finalizing the changes by flipping that extra byte noice

>How does SQLite really create a database and interact with it? through regedit

How do file formats work. In windows they are a file extension in Unix they are a magic number that indicating the file type. >But how does sqlite make a database out of a file There is no uniform answer to how a file does something. Files are just 1's and 0's representing data that follow some set of rules set by a standard. A program reads in a file and knows from the magic number or extension the type of file and if it was programmed to handle that type of file handles it according to its rules. I mean think about ASCII or UTF-8. The data in the file has different meaning based on the the choice of encoding. Once you handle the encoding you might have the choice between XML and json. These two formats might be text based, but the rules around valid text is different depending on which format it actually is. This doesn't even include things like formats that just exist for a specific program.

>>37480 emacs can also do this with hexl-mode, and you can edit it

dumdums cant into not overwriting important data so they decided that they would rather their process die on a segmentation fault. Basically the deal with file formats is that you want to have some parts of the file (data and code) be separate so that they will get mapped with different memory attributes. Then you know this breeds the need for relocation of pointers inside of the file, for which there are relocation tables with relocation entries.

>>32542 >how do programs actually work with binary files and how do file formats work? Unix systems (and others) look for particular byte code patterns to determine the file format, these are called "magic patterns", or commonly referred to as the "file magic". $ man magic >>37480 >No it's just hexyl, a tool to print a file's data in hex: >>39673 >emacs can also do this with hexl-mode, and you can edit it As does vi. Toggle hex modes with (if installed obviously) :%!xxd and :%!xxd -r : - the following is a command to run % - select all of file ! - replace selected with xxd - run xxd with stdin (=all of file contents in this case) and output to stdout. -r - do in reverse od, hd, hexdump may also prove useful to know, and are likely installed by default, but then reversal generally requires a script or separate utiility unhexdump, hex2bin, bin2hex - so easier to use xxd for both. binwalk is very useful further down this rabbit hole

>>40016 Hilariously accurate take. >PLT? GOT? Nonsense dreamed up by the utterly deranged! >30 years of dynamic libraries and no real world usage found