So I run a video production company. We have 300TB of archived projects (and growing daily).

Many years ago, our old solution for archiving was simply to dump old projects off onto an external drive, duplicate that, and have one drive at the office, one offsite elsewhere. This was ok, but not ideal. Relatively expensive per TB, and just a shit ton of physical drives.

A few years ago, we had an unlimited Google Drive and 1000/1000 fibre internet. So we moved to a system where we would drop a project onto an external drive, keep that offsite, and have a duplicate of it uploaded to Google Drive. This worked ok until we reached a hidden file number limit on Google Drive. Then they removed the unlimited sizing of Google Drive accounts completely. So that was a dead end.

So then we moved that system to Dropbox a couple of years ago, as they were offering an unlimited account. This was the perfect situation. Dropbox was feature rich, fast, integrated beautifully into finder/explorer and just a great solution all round. It meant it was easy to give clients access to old data directly if they needed, etc. Anyway, as you all know, that gravy train has come to an end recently, and we now have 12 months grace with out storage on there before we have to have this sorted back to another sytem.

Our options seem to be:

  • Go back to our old system of duplicated external drives, with one living offsite. We’d need ~$7500AUD worth of new drives to duplicate what we currently have.
  • Buy a couple of LTO-9 tape drives (2 offices in different cities) and keep one copy on an external drive and one copy on a tape archive. This would be ~$20000AUD of hardware upfront + media costs of ~$2000AUD (assuming we’d get maybe 30TB per tape on the 18TB raw LTO 9 tapes). So more expensive upfront but would maybe pay off eventually?
  • Build a linustechtips style beast of a NAS. Raw drive cost would be similar to the external drives, but would have the advantage of being accessible remotely. Would then need to spend $5000-10000AUD on the actual hardware on top of the drives. Also have the problem of ever growing storage needs. This solution we could potentially not duplicate the data to external drives though and live with RAID as only form of redundancy…
  • Another clour storage service? Anything fast and decent enough that comes at a reasonable cost?

Any advice here would be appreciated!

  • alonesomestreet@alien.topBEnglish
    1·
    1 year ago

    How often are you actually needing to access the 300TB? If 250TB are “cold storage”, then LTO is the way (you can rent the readers usually, rather than buy)

    If you’re needing to have access but not edit from, NAS is the way, 300TB wouldn’t even be THAT expensive (still expensive), just slow to move to, but once you’re up and running a decent rig should last years.

    If you’re needing to access all 300TB, then you’re looking at a LTT style NAS that needs to handle read and write from multiple users at a time, and that’s gonna be the real $$$.

    I feel like you might do well from a mixture of all of these. A smallish NAS for day to day/project use, and once that project is done you move it to the big “slow” server for onsite backup, and once every 2-3mo you rent the LTO drive and load up a few tapes, and ship them off to the void for offsite backup and cold storage.

    • campster123@alien.topOPBEnglish
      1·
      1 year ago

      Just copying from a response above:

      This is only for archived projects. But we’d probably still need to access ~10-20TB of that data relatively regualry to update branding, or change edits, etc. Saying that, as mentioned in the OP, if we went tape or cloud, we’ll likely have a physical local copy on an external hard drive for quicker access. We just need a redunant back up of these archives.

      If we went NAS, I feel like maybe we could get away without the redundancy? Risky…

      • alonesomestreet@alien.topBEnglish
        1·
        1 year ago

        I mean; the NAS would have some built in redundancy via RAID 5 or 6 or whatever, but you wouldn’t have an offsite backup. What you’d wanna look into is something like Backblaze B2, but even that is going to be $1800 a month, so at that point I would say build a 2nd NAS and pay for it to be in a data center, that would only be a couple hundred a year, or even just run it at your house and run a nightly backup.

      • cosmin_c@alien.topBEnglish
        1·
        1 year ago

        If we went NAS, I feel like maybe we could get away without the redundancy? Risky…

        That’s the thing, you could, but it wouldn’t be best practice. At the end of the day the 3-2-1 rule applies to any data.

        I know it’s a hard pill to swallow, but ideally you’d need both a NAS (I’d go with Proxmox on a PC) and the tape backup for that NAS to ensure the safety of the data.

        However. Backblaze may take the spot of the tapes - unsure if the NAS as well. Have a look at their offer and see what fits your budget. I would personally go with the NAS on site and backup it daily to Backblaze. Note that Backblaze B2 says something like 6$/TB/month which amounts to about 21600$/year which stings but then again it’s safe and it’s the best value (all the competition seems to be more expensive).