VidCompiler
Turns any file into a video and hands back one link. Download it later and every byte comes back exactly as it was.
How it works
- 01Archivetar.gz, skipped when it would not shrink
- 02EncryptAES-256-GCM, scrypt key
- 03ProtectRS(255,215): 40 parity per 215 bytes
- 04Planes2 bits a block on brightness, 3 on each colour plane
- 05Tonesbackup header as 4-FSK audio, 2 bits a tone
- 06EncodeH.264, hardware where there is any
- 07Shardssliced and uploaded concurrently, 2.14x
- 08One linkevery shard in one unlisted playlist
Each panel draws what that step does to the data. The interesting ones are the middle three: the payload goes into the three colour planes at two different densities, a copy of the header is modulated onto the audio track as four tones, and the shards upload at the same time rather than one after another.
A file becoming a frame
Draw on the pad — that drawing is the file. The read head takes it a byte at a time, each byte is encrypted, and the bits are written into the three planes a video frame is actually made of: the large grey one is brightness at 2 bits a block, and the two smaller coloured ones are Cb and Cr at 3 bits a block. They are quarter size because 4:2:0 stores colour at half the resolution in each direction, and each block is drawn as the colour that chroma value really means — the colour of a block is the three bits stored in it. The frame looks like static because it should: what reaches the pixels is ciphertext, and a frame you could read the drawing off would mean nothing had been encrypted. Add a stroke and watch the blocks for those bytes, and only those, come out different. One padful fills all three planes exactly; a real frame is 480×270 luma blocks holding 56,100 bytes.
Video hosting is free and unmetered; file hosting is not. So the file becomes the video. The archive is packed, encrypted, error-corrected, written into the pixels of 1080p frames, encoded to H.264 and uploaded unlisted. Decoding runs the same road backwards and checks the result against a CRC. On v7 the whole thing comes back as a single playlist link, and a file manager sits on top of that, so it behaves like a drive rather than like a tool.
A frame is not one surface but three. H.264 keeps brightness and colour in separate planes at different resolutions, and probe uploads through real YouTube found they survive very differently: colour came back bit-perfect at every density tried, while brightness broke between 2 and 3 bits per block. That is exactly why the previous version failed — 3 bits per block on the luma plane. So the format writes 2 bits per 4x4 block of brightness and 3 bits per 4x4 block on each of the two colour planes, which is 56,100 bytes to a frame at a measured bit error rate of 1.1e-05. What luma needs turned out to be contrast rather than area: 2x2 blocks of pure black and white survived perfectly where 4x4 blocks of eight greys did not.
Reed-Solomon RS(255,215) covers what the transcode still damages: every 215 bytes of payload carries 40 bytes of parity, enough to rebuild any 20 broken bytes in the block without knowing in advance which ones broke.
The header is written three independent ways, because losing it loses everything. It goes into the pixels five times over, into the video description as base64 — the one channel YouTube stores losslessly — and into the audio track as 4-FSK tones. Four tones means every symbol carries two bits, and 3,150 symbols a second is the bandwidth-limited optimum: tones must be spaced at least a symbol rate apart to stay orthogonal, AAC low-passes around 15 kHz, and one symbol rate higher pushes the top tone past that cutoff. Sweeping every viable arrangement through a real AAC round trip put 4-FSK at 652 payload bytes a second, 2.9x the binary setting it replaced, and 8-FSK behind it. Against the pixel channel's 1.9 MB/s that is 0.01% of capacity — it is redundancy, not throughput, and it earns its place by being a completely different failure mode.
Anything past about 1.27 GB does not fit in one video, so archives shard. The reason to do that is upload: a single HTTP stream does not saturate the link, and YouTube throttles per stream rather than per connection. Three concurrent uploads measured 39.9 Mbit/s against 18.6 for one — 2.14x — even though each individual stream dropped to 13.4. Download parallelises harder still, at 189.8 Mbit/s across three. Sharding the local compute barely helps by comparison, 1.26x, because the pixel and Reed-Solomon stages already use every core and consumer NVENC caps concurrent sessions.
v7 is about what you are left holding. A sharded archive used to hand back a list of URLs, every one of which was required, and the list itself became the thing you could lose. v7 collects the videos into an unlisted playlist and returns one link; the playlist records the order, which is what the decoder needs anyway. Reading it back uses yt-dlp rather than the API, so decoding needs no credentials at all — an unlisted playlist is readable by link, and demanding OAuth to read something the link already grants would be a poor trade. Two smaller things landed with it: the archiver samples a couple of megabytes and skips gzip when compression will not help, taking incompressible input from 31.9 to 190.9 MB/s, and uploads now start as each shard finishes encoding rather than after all of them, so the first one begins around 25 seconds in instead of 95.
Everything is sealed with AES-256-GCM under a scrypt-derived key before it leaves the machine, so what YouTube stores is neither your contents nor your filenames.
Highlights
- One playlist link per archive, however many videos it took
- Format picked from real round trips, not from theory
- Brightness and colour planes carry different densities, by measurement
- Header carried three ways: pixels, 4-FSK audio, description
- Concurrent shard upload, measured at 2.14x
- Verified byte-identical from 10 MB to 5 GB
- Reed-Solomon and the pixel engine in native C, multithreaded
- AES-256-GCM before anything leaves the machine
Built with
- Python
- C
- NumPy
- Flask
- FFmpeg
- Reed-Solomon
- 4-FSK
- AES-256-GCM
- Docker
- YouTube API