Anyone tried PuralEyes - or similar?

[dave2010]

I stumbled across mention of PluralEyes - http://www.redgiant.com/products/pluraleyes/ quite by accident - by way of looking for a reasonable, but not hugely expensive portable music player. After going round several houses, I found a recommendation for a voice recorder - a Sony PCM-M10, and in one of the pages relevant to that device there was mention of PluralEyes - see Ken Rockwell's review - http://www.kenrockwell.com/audio/sony/pcm-m10.htm

Potentially, if it works reasonably well, it could be very helpful to some - but of course of no use to almost everyone else who is not involved in making video or audo productions. This could solve or avoid a potential problem for anyone making video recordings in the post production stages. Most people recording on a single video camera would not have any problems, and even if for some reason audio and video went out of sync, there should only be one adjustment needed. However, if a recording is made using multiple devices, each with its own audio channels, then the situation could arise in which it would be useful to use different video clips from the different devices, but it might be crucially important to have the music or sound tracks seamlessly and consistently linked to the video. This would be important for video recordings of concerts, where almost any timing glitches in the audio tracks would be noticeable.

An approach to recording could also use audio only inputs added to the mix of video sources - for example if a high quality stand alone audio recorder, such as the Sony PCM-M10 is used to record the sound track, and this is then used to replace the sound track from any of the video cameras used at the performance. There might also be additional audio sources, such as portable devices used to get close up sound - perhaps particularly important for soloists, or which might have been used as backup recorders during the performance.

I had already envisaged such a scenario, and it is relevant to the separate thread on live recording - http://theartofsound.net/forum/showthread.php?40936-Live-recording What I hadn't realised is that there is equipment and software to do this, and presumably production teams which use these methods. Ideally, as with a lot of media work, it would be better to avoid problems altogether, and get perfect data capture (noth audio and video) and have everything synced from the start, but this is not always possible and if tools such as PluralEyes work well, then for some applications they could give a very much improved result.

I had alrady figured out that except for face close-ups, that there could be a little leeway in syncing audio and video clips, for example only a very picky musician might notice a fraction of a second discrepancy between sound and video in a clip showing an orchestral conductor doing a downbeat, or an instrumentalist playing a solo passage. Discrepancies between audio and video are very obvious for speech or singing - lip sync really is important. If software such as PluralEyes works well, such problems could largely be eliminated. Also, if it is easy enough to use, it could enable more complex and interesting multimedia productions.

Has anyone used software like this, or worked in a production requiring multiple input sources and careful syncing of sound and video?

Comments?

[Rothchild]

I'm sure there's something I'm missing (the website isn't totally descriptive) but this is 'just' syncing the audio?

If so doesn't that requires that the audio is already synced to the picture? I suppose if you've got a capture from the camera and an external recorder it will help you with that but it seems a rather expensive way around having someone simply clap (or use a clapboard) when you start rolling and then lining up that transient in picture and sound.

Disclaimer - I'm a sound guy not a video guy so I know very little of the conventions and techniques used there, but I have done some shooting and editing of video.

[dave2010]

I'm sure there's something I'm missing (the website isn't totally descriptive) but this is 'just' syncing the audio?

If so doesn't that requires that the audio is already synced to the picture? I suppose if you've got a capture from the camera and an external recorder it will help you with that but it seems a rather expensive way around having someone simply clap (or use a clapboard) when you start rolling and then lining up that transient in picture and sound.

Disclaimer - I'm a sound guy not a video guy so I know very little of the conventions and techniques used there, but I have done some shooting and editing of video.The point is that if you have multiple inputs - including both video and audio, they can be synced. Also if separate clips are generated during an editing process, then this tool - if it works well enough - would keep synchronisation reasonably well. Whether it would be more important to sync the audio or the video would probably depend on the application, though for what I have in mind the audio is more important as even small glitches would be obvious. There are tricks which could be used with video across transitions, or even putting in some still material, maybe with a "Ken Burns" technique, which could make problems with the video result less obvious.

Of course there may also be some problems in the audio, and having alternative input sources for those might also help, though whether that would be easy to synchronise using the software mentioned here I'm not sure. Personally I'd load the two audio sources into audacity, then loop them, and gradually shift one relative to the other until they were in sync, but I do think that in some ways audio is easier to deal with. I have tried this before, to patch in a short section of a recording which was obtained by a different route, but it did turn out to be trickier than I'd thought it would be, and I wasn't sure even then if the joins were sufficiently good enough to be inaudible.

[Rothchild]

The point is that if you have multiple inputs - including both video and audio, they can be synced.

This is my confusion, it seems to suggest that all the clips need audio on them, it's not syncing audio to video just audio to audio (for which the mark one eyeball and earhole work with astounding accuracy - especially on a hard transient like a hand clap or clapper board)

It seems like a product that bridges a gap that I didn't realise people face, either you shoot silent with a separate recorder and use a clapper board or you're a bit further up the food chain and you sync multiple cameras and print timecode with your shoot.

But as I said I don't really know about pictures. On the other hand if you want a hand with audio editing and compilation I'd be more than happy to help (and I know a fair bit about that).

[dave2010]

It seems like a product that bridges a gap that I didn't realise people face, either you shoot silent with a separate recorder and use a clapper board or you're a bit further up the food chain and you sync multiple cameras and print timecode with your shoot.
Re clapper boards etc. - that'd be fine for some recordings, but for a live event, it's not going to work - not unless it's a very odd event. The one I was involved in - I was invited to record audio - then video was sneaked in. It was in a church, so I don't think clapper boards would have been appreciated. I've learned a lot - not the least thing is perhaps not to try in the first place!

An audio CD has been made, and is apparently satisfactory. The DVD is emerging slowly. I did manage to make a DVD a couple of days ago, and it plays OK on my Blu Ray and TV, though I'd really like to tweak it, and hopefully produce something much better.

However, I'm getting there - slowly!

[Rothchild]

Yeah, I know it's a project that's 'evolved' for you. Whilst I appreciate that a proper clapperboard may not be appropriate for the context the point I'm trying to make is that (to my basic understanding) is that video audio sync (for a simple setup) requires nothing more than a clear visual and audio cue (someone clapping their hands together once would do it). I know you also had issues with the capacity of the recorder etc, but if you had enough storage you could start the recording (of sound and video) quite a bit before the performance, put the clap/cue in and leave it all rolling.

[Jason P]

PluralEyes works well, depending on the edit software you use. It's always been possible (but laborious) to manually phase-sync audio by ear, where program's like PE come in to their own is when you've footage that stops/starts so there are multiple sync points.

However, some of the newer editing programs have reduced the need for this. Final Cut Pro X has excellent sync capabilities built in, working just as Plural Eyes does by analysing and matching waveforms. Of course this does mean you have to have recorded audio on all your sources - even if it's only the on-camera mike for the video.

[Audio Advent]

Clapping is normally the first thing that happens at a concert as musicians populate the stage - just make a single clap yourself in front of the camera.... although that would have to be near the mic so depends what you will have done there. Still, if there's a conductor involved there could well be a tap on the lecturn with their baton.

[dave2010]

I went looking for details of Adobe video editing packages today, and I found this review of Premiere Pro CC - http://professional-video-editing-software-review.toptenreviews.com/adobe-premiere-pro-review.html

It looks as though audio synchronisation is a feature of that package - and this is also recognised as a potential problem area.
Hopefully that package overcomes audio synchronisation problems, though it might be a bit expensive for casual/amateur users.

[Rothchild]

I saw this and it made me think of this thread:

<iframe width="560" height="315" src="https://www.youtube.com/embed/Ul04AA3R4d0" frameborder="0" allowfullscreen></iframe>

Linky in case embedding doesn't work: https://www.youtube.com/watch?v=Ul04AA3R4d0

[dave2010]

I saw this and it made me think of this thread:

<iframe width="560" height="315" src="https://www.youtube.com/embed/Ul04AA3R4d0" frameborder="0" allowfullscreen></iframe>

Linky in case embedding doesn't work: https://www.youtube.com/watch?v=Ul04AA3R4d0
Fine for films, I guess. Doesn't go down so well at public events, such as theatres, music in concert halls, or concerts in churches.

I'm gradually figuring out ways to do audio and video synchronisation, though getting it very accurate is hard. Mostly mayb e it doesn't matter to within 100 ms or so, though if video includes speech then lip sync errors can be detected with an "error" of 10 ms or more. Probably watching wind soloists playing and syncing music wouldn't be too critical to within one or two frames. Not so sure about pianists - I think some "error" could be tolerated. Watching string players, such as violinists play rapid passages it is clear that even small errors are noticeable if the audio isn't well enough synchronised. I think that such sequences are probably about as demanding as speech.

Not sure about watching percussionists.

Also remember that we must be used to timing errors. Sit in the front row of a concert hall, and the performers are perhaps a minimum of 15 feet away, which is around 13 ms delay between "video" reception and aural stimulus at the ear. Sit at the back, or in the balcony area, but still within sight of the performers and 100 ms is easily possible if the performers are 100 ft or more away, yet mostly we don't notice. One conductor, Sir Adrian Boult, used to ask players at the rear of the orchestra to play right on his beat or perhaps a fraction before it. HIs explanation for that was that if there is (say) 40 feet between the front and back of a large orchestra in a concert hall, then if the players follow the method of listening to each other, to keep time then it doesn't work properly. The violins being at the front could follow the conductor's beat fairly accurately, but if the players at the back use listening cues from the strings then that could give a lag of 30ms or more before they even play. As the sound then has to travel back towards the front, then the effective synchronisation between the strings at the front, and the wind and brass players at the back could quite easily have a lag of 60ms up to 100 ms. This, he claimed, would be noticeable to the audience in the seated area in the concert hall.

I have never been fully convinced of this theory though, as players who listen closely to each other do often give better performances.

[Rothchild]

Yeah, calling '28 prick' might be a little embarrassing, but as far as tools to sync sound and picture (and to leave clues for the editors as to what's going on) it's about as KISS as one can get.

In terms of an orchestra adjusting for the 'latency' of the distance between them, well, the speed of sound is not a theory but how you get it to map out practically is a bit more complicated than saying 'guys at front play half a beat late' as it's going to depend on the tempo of the piece (and it's not so easy to say to the guys at the front 'play xms late').

I've noticed plenty of tv documentaries where various relatively random footage has been lined up with non matching audio (gigs against records and the like) it seems to me that as long as the next cut is just at or around the point that it becomes obvious most viewer will let it go. When recording audio with latency most capable performers can cope with up to around 7ms quite easily (although obviously less is always better). I think orchestras are a fairly singular example as many other types of bands and groups would have some sort of monitoring setup so the sound from the instrument on the other side of the stage is being 'beamed' electronically through the monitoring rather than on the air across the stage.