Upstreaming Facebook extensions

[Matt Harbison]

> On Nov 16, 2017, at 2:03 PM, Augie Fackler <raf at durin42.com> wrote:
> 
> 
>> On Nov 16, 2017, at 12:05, David Soria Parra <david.soriaparra at oculus.com> wrote:
>> 
>> Already commented on this. We use this day-to-day and it seems to be solid. A few things:
>> 
>> - UI:
>> - The progress bar gets confused by downloads, as those are done one by one. We need to improve how we show the amount of lfs files and the bytes to download. Something we can figure out while upstreaming
>> - Resumable Downloads:
>> - We don’t support that yet. Should be in there, but not urgent for upstreaming.
> 
> None of these seem worrying
> 
>> - Additional consistency checks:
>> - We know that somehow it can happen that a file only gets partially downloaded. We need to better consistency check before upstreaming.
> 
> seems vaguely important for a "done" feature

I couldn’t interrupt the download, but if I corrupt the file in the local store and then update/cat/etc, that aborts with a message about corruption.  So it’s not silent corruption, and may be just a case of figuring out how to fix the file in the local store.  I’m not sure largefiles handles that case any better.

That said, I see it will happily upload a corrupt file.  I’ve got a patch once the first series is queued.

>> - Interaction with largefiles
>> - We need a consistent story how to get from largefiles to LFS and back. Alternatively we can just say “you can’t use LFS if you used largefiles”.
> 
> I'm fine to just have the extension refuse, and point to docs on how to use convert to escape the clutches of the old largefiles. Not sure if that's practical, but it's at least a  start and we could advance the state of the art later if it proves painful for people.

I haven’t tried with largefiles yet, but I wonder if that works ok because it doesn’t actually track the huge file.  I’ve seen code that attempts to handle switching between large and normal, so switching might just be a remove and re-add.  The only thing that seems iffy is the config settings that allow *new* files to be added as large automatically.  It would be nice if they played together with convert.