There is magic in the files...

Back in olden days, we used to use a command called “file” in *nix and related systems. While the upstart OS2, DOS and Windows operating systems encouraged file extensions by design, *nix always let the user decide what to name each file. A file name ending with “.exe” need not mean it can/cannot be executed. Given the more heterogenous environments and files being copied back-n-forth with total disregard, we had to check whether the file is really what it seems to be… To take a trivial case, text files copied back and forth between DOS and *nix…

$ file README
README: ASCII Text

$ file OTHERFILE.TXT
OTHERFILE.TXT: ASCII text, with CRLF line terminators

The file command was magical. It could not only tell you a file contains text, but whether it is shell script of a particular type etc. If it was an executable/binary, how it was linked, what target architecture it was meant for etc.

What made the command so smart? Turns out that among other things it uses the concept of “Magic Numbers”. This is a frequently used technique that involves adding a header section to a file or block to identify the type of the file/data. By reading the first few bytes, if not a bit more, we can guess what a file is meant to contain.

So, imagine my surprise when reading up on “BitCoin” and the underlying block-chains. As the name suggests, BitCoin and other crypto currencies that rely on distributed ledger mechanisms, use ‘blocks’ of data. And lo and behold, the first 4 bytes of each block is always “0xD9B4BEF9”. Does that mean anything?

In the early days, the magic numbers were selected by geeks to represent something meaningful to them. Here is a sampling …

Even now-a-days, I occasionlly have to check the leading part of some files when I encounter a corrupt PDF (maybe it is a zip/rar or a postscript file…).