So you have a some h264 NALUs (eg from an IP camera over RTSP), and you want to save them to an mp4 file. The most sensible way to do this seems to be via avcodec (which is part of ffmpeg). fmpeg is an incredible tool, but there’s not a lot in the way of documentation, especially when it comes to specific codecs. Saving your video content this way is beneficial because you don’t have to perform any re-encoding.

This took me a series of trial and error attempts, so I thought I’d write it here in case anybody else encounters the same question.

I’ll just get straight to the point:

• The first packet must be a combination of SPS, PPS, and IDR NALUs, all joined together.

That’s pretty much all there is to it. You just concatenate the 3 NALUs into one buffer and call av_write_frame.

Thereafter, write your subsequent keyframes or intermediate frames directly. I’m not sure whether mp4 supports changing SPS or PPS midstream, and in my particular use case, I don’t need that, so I discard all subsequent SPS and PPS NALUs and only write visual NALUs.

You’ll need to encode the NALUs in annex-b format. This stackoverflow post is an excellent overview of annex-b and some other h264 encoding fundamentals.