DigitalOcean Spaces, S3QL and Mount Folder on Linux Server

I usually use Amazon S3 to store files that are rarely accessed or “infrequent access” but the file itself still needed and should not be deleted. So it’s a kind of storage tier model, where the main storage uses SSD hard drive, the second disk uses block storage and the third disk uses Amazon S3.

Storage tier-1 is used for primary storage, tier-2 is used for replica data or slave data or non-transactional data while tier-3 is used for archival or backup data or data that is rarely accessed.
On a Linux system, I tend to use S3ql to mount Amazon S3 data. Because DigitalOcean (DO) has released S3-alike services for a long time, namely DO Spaces, I try to use it by making DO spaces as additional folders on Linux servers. The mounting process is done using S3ql.
Below is how to create DO spaces and mount it on Linux Serve using S3ql :

1. 1. Create 1 DO spaces bucket and select the nearest region. This process just similar as if using AWS S3. If you haven’t used DO or don’t have a DO account, you can use my referral link to get an additional $ voucher : DO Signup.
      
   2. Create a bucket name that matches its function and is easy to remember. Don’t forget to choose the appropriate option (for example activating CDN so that it is easily accessible in different geographical locations, disable listing files etc.)
   3. Click on the side menu, select the API menu. In the API view, select the Spaces access keys section, create AccessKey and Secret (password) by clicking on the “Generate New Key” button
   4. Write down the bucket name that has been created, Access Key and its Secret / Password
   5. On the Linux server that you want to use, install the s3ql package. On Ubuntu, just run the command: :
      [code lang=”bash”]
      apt-get install s3ql
      [/code]
   6. Run the following command to process the partition as well as mount the folder (eg mounted on the /srv/vavai folder using a bucket with the name excellent-vavai) :
      [code lang=”bash”]
      mkdir /srv/vavai
      mkfs.s3ql –plain –cachedir /var/cache/s3ql s3c://sgp1.digitaloceanspaces.com:443/excellent-vavai/
      mount.s3ql –cachedir /var/cache/s3ql s3c://excellent-vavai.sgp1.digitaloceanspaces.com:443/excellent-vavai /srv/vavai/ –allow-other –cachesize 1000000
      [/code]
   7. When running the above command, you will usually be asked for a login back-end and back-end password. The login back-end is AccessKey while the back-end password is SecretKey. If you don’t want to repeating AccessKey and Password, we can create ~/.s3ql/authinfo2 files (for example /root/.s3ql/authinfo2 or /home/vavai/.s3ql/authinfo2 depending on the user’s home folder) with the following contents :
      [code lang=”bash”]
      storage-url: s3c://vavai-excellent.sgp1.digitaloceanspaces.com
      backend-login: YourAccessKey
      backend-password: YourSecret
      fs-passphrase: passphrase-to-encrypt
      [/code]
      Then run the mkfs command and mount it above with the additional parameter –authfile /root/.s3ql/authinfo
   8. After all these steps, check with the df -h command, the mount folder should already exist. We can do the testing the folder by creating or copying a file to the mount folder and then checking its contents. s3ql automatically performs the data encryption process, so that what appears next to the DO spaces is the encrypted files, but if it is checked in the mount folder, the result is the normal files and folders as they should
      

With the cost of DO spaces are relatively low compared to Amazon S3 or other object storage providers, we can choose DO spaces as an alternate infrequent access storage.