So after a bunch of experiments I arrived at a working sequence that I think is a good proof of concept. The ‘proper’ end version will be waaaaay slower and likely have some video superimposed but for now this is worth sharing I think.
Thanks LN and Sam Ballard for useful feedback.