<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>S3 Archives - Vavai&#039;s Personal Notes</title>
	<atom:link href="https://www.vavai.net/tag/s3/feed/" rel="self" type="application/rss+xml" />
	<link>https://vavai.net/tag/s3/</link>
	<description>Life is our precious gift</description>
	<lastBuildDate>Fri, 19 Oct 2018 03:47:58 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	
	<item>
		<title>DigitalOcean Spaces, S3QL and Mount Folder on Linux Server</title>
		<link>https://www.vavai.net/digitalocean-spaces-s3ql-and-mount-folder-on-linux-server/</link>
					<comments>https://www.vavai.net/digitalocean-spaces-s3ql-and-mount-folder-on-linux-server/#respond</comments>
		
		<dc:creator><![CDATA[Muhammad Rivai]]></dc:creator>
		<pubDate>Fri, 19 Oct 2018 03:47:58 +0000</pubDate>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[Amazon]]></category>
		<category><![CDATA[DigitalOcean]]></category>
		<category><![CDATA[Object Storage]]></category>
		<category><![CDATA[S3]]></category>
		<category><![CDATA[Spaces]]></category>
		<guid isPermaLink="false">http://v2.vavai.net/?p=2040</guid>

					<description><![CDATA[<p>I usually use Amazon S3 to store files that are rarely accessed or &#8220;infrequent access&#8221; but the file itself still needed and should not be deleted. So it&#8217;s a kind of storage tier model, where the main storage uses SSD hard drive, the second disk uses block storage and the third disk uses Amazon S3....</p>
<p>The post <a href="https://www.vavai.net/digitalocean-spaces-s3ql-and-mount-folder-on-linux-server/">DigitalOcean Spaces, S3QL and Mount Folder on Linux Server</a> appeared first on <a href="https://www.vavai.net">Vavai&#039;s Personal Notes</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>I usually use Amazon S3 to store files that are rarely accessed or &#8220;infrequent access&#8221; but the file itself still needed and should not be deleted. So it&#8217;s a kind of storage tier model, where the main storage uses SSD hard drive, the second disk uses block storage and the third disk uses Amazon S3.<br />
<img fetchpriority="high" decoding="async" class="alignnone size-full wp-image-2042" src="http://54.254.121.108/wp-content/uploads/2018/10/do-spaces.png" alt="" width="2496" height="1064" srcset="https://d3b35uem336dgh.cloudfront.net/wp-content/uploads/2018/10/do-spaces.png 2496w, https://d3b35uem336dgh.cloudfront.net/wp-content/uploads/2018/10/do-spaces-300x128.png 300w, https://d3b35uem336dgh.cloudfront.net/wp-content/uploads/2018/10/do-spaces-1024x437.png 1024w, https://d3b35uem336dgh.cloudfront.net/wp-content/uploads/2018/10/do-spaces-768x327.png 768w, https://d3b35uem336dgh.cloudfront.net/wp-content/uploads/2018/10/do-spaces-1536x655.png 1536w, https://d3b35uem336dgh.cloudfront.net/wp-content/uploads/2018/10/do-spaces-2048x873.png 2048w" sizes="(max-width: 2496px) 100vw, 2496px" /><br />
Storage tier-1 is used for primary storage, tier-2 is used for replica data or slave data or non-transactional data while tier-3 is used for archival or backup data or data that is rarely accessed.<br />
On a Linux system, I tend to use S3ql to mount Amazon S3 data. Because DigitalOcean (DO) has released S3-alike services for a long time, namely DO Spaces, I try to use it by making DO spaces as additional folders on Linux servers. The mounting process is done using S3ql.<br />
Below is how to create DO spaces and mount it on Linux Serve using S3ql :</p>
<ol>
<li style="list-style-type: none">
<ol>
<li>Create 1 DO spaces bucket and select the nearest region. This process just similar as if using AWS S3. If you haven&#8217;t used DO or don&#8217;t have a DO account, you can use my referral link to get an additional $ voucher : <a href="https://m.do.co/c/33ab1b84b144">DO Signup</a>.<br />
<img decoding="async" class="alignnone size-full wp-image-9100" src="https://www.vavai.com/wp-content/uploads/2018/10/Screenshot-2018-10-14-19.29.38.png" alt="" width="2182" height="1274" /></li>
<li>Create a bucket name that matches its function and is easy to remember. Don&#8217;t forget to choose the appropriate option (for example activating CDN so that it is easily accessible in different geographical locations, disable listing files etc.)</li>
<li>Click on the side menu, select the <strong>API</strong> menu. In the API view, select the Spaces access keys section, create AccessKey and Secret (password) by clicking on the <strong>&#8220;Generate New Key&#8221;</strong> button</li>
<li>Write down the bucket name that has been created, Access Key and its Secret / Password</li>
<li>On the Linux server that you want to use, install the s3ql package. On Ubuntu, just run the command: :<br />
[code lang=&#8221;bash&#8221;]<br />
apt-get install s3ql<br />
[/code]</li>
<li>Run the following command to process the partition as well as mount the folder (eg mounted on the /srv/vavai folder using a bucket with the name excellent-vavai) :<br />
[code lang=&#8221;bash&#8221;]<br />
mkdir /srv/vavai<br />
mkfs.s3ql &#8211;plain &#8211;cachedir /var/cache/s3ql s3c://sgp1.digitaloceanspaces.com:443/excellent-vavai/<br />
mount.s3ql &#8211;cachedir /var/cache/s3ql s3c://excellent-vavai.sgp1.digitaloceanspaces.com:443/excellent-vavai /srv/vavai/ &#8211;allow-other &#8211;cachesize 1000000<br />
[/code]</li>
<li>When running the above command, you will usually be asked for a login back-end and back-end password. The login back-end is AccessKey while the back-end password is SecretKey. If you don&#8217;t want to repeating AccessKey and Password, we can create ~/.s3ql/authinfo2 files (for example /root/.s3ql/authinfo2 or /home/vavai/.s3ql/authinfo2 depending on the user&#8217;s home folder) with the following contents :<br />
[code lang=&#8221;bash&#8221;]<br />
storage-url: s3c://vavai-excellent.sgp1.digitaloceanspaces.com<br />
backend-login: YourAccessKey<br />
backend-password: YourSecret<br />
fs-passphrase: passphrase-to-encrypt<br />
[/code]<br />
Then run the mkfs command and mount it above with the additional parameter <strong>&#8211;authfile /root/.s3ql/authinfo</strong></li>
<li>After all these steps, check with the df -h command, the mount folder should already exist. We can do the testing the folder by creating or copying a file to the mount folder and then checking its contents. s3ql automatically performs the data encryption process, so that what appears next to the DO spaces is the encrypted files, but if it is checked in the mount folder, the result is the normal files and folders as they should<br />
<img decoding="async" class="alignnone size-full wp-image-9101" src="https://www.vavai.com/wp-content/uploads/2018/10/Screenshot-2018-10-14-20.29.16.png" alt="" width="2042" height="1022" /></li>
</ol>
</li>
</ol>
<p>With the cost of DO spaces are relatively low compared to Amazon S3 or other object storage providers, we can choose DO spaces as an alternate infrequent access storage.</p>
<p>The post <a href="https://www.vavai.net/digitalocean-spaces-s3ql-and-mount-folder-on-linux-server/">DigitalOcean Spaces, S3QL and Mount Folder on Linux Server</a> appeared first on <a href="https://www.vavai.net">Vavai&#039;s Personal Notes</a>.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://www.vavai.net/digitalocean-spaces-s3ql-and-mount-folder-on-linux-server/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
		<item>
		<title>DigitalOcean Spaces, S3QL and Mount Folder on Linux Server</title>
		<link>https://www.vavai.net/digitalocean-spaces-s3ql-and-mount-folder-on-linux-server-2/</link>
					<comments>https://www.vavai.net/digitalocean-spaces-s3ql-and-mount-folder-on-linux-server-2/#respond</comments>
		
		<dc:creator><![CDATA[Muhammad Rivai]]></dc:creator>
		<pubDate>Fri, 19 Oct 2018 03:47:58 +0000</pubDate>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[Amazon]]></category>
		<category><![CDATA[DigitalOcean]]></category>
		<category><![CDATA[Object Storage]]></category>
		<category><![CDATA[S3]]></category>
		<category><![CDATA[Spaces]]></category>
		<guid isPermaLink="false">http://v2.vavai.net/?p=2040</guid>

					<description><![CDATA[<p>I usually use Amazon S3 to store files that are rarely accessed or &#8220;infrequent access&#8221; but the file itself still needed and should not be deleted. So it&#8217;s a kind of storage tier model, where the main storage uses SSD hard drive, the second disk uses block storage and the third disk uses Amazon S3....</p>
<p>The post <a href="https://www.vavai.net/digitalocean-spaces-s3ql-and-mount-folder-on-linux-server-2/">DigitalOcean Spaces, S3QL and Mount Folder on Linux Server</a> appeared first on <a href="https://www.vavai.net">Vavai&#039;s Personal Notes</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>I usually use Amazon S3 to store files that are rarely accessed or &#8220;infrequent access&#8221; but the file itself still needed and should not be deleted. So it&#8217;s a kind of storage tier model, where the main storage uses SSD hard drive, the second disk uses block storage and the third disk uses Amazon S3.<br />
<img loading="lazy" decoding="async" class="alignnone size-full wp-image-2042" src="http://54.254.121.108/wp-content/uploads/2018/10/do-spaces.png" alt="" width="2496" height="1064" srcset="https://d3b35uem336dgh.cloudfront.net/wp-content/uploads/2018/10/do-spaces.png 2496w, https://d3b35uem336dgh.cloudfront.net/wp-content/uploads/2018/10/do-spaces-300x128.png 300w, https://d3b35uem336dgh.cloudfront.net/wp-content/uploads/2018/10/do-spaces-1024x437.png 1024w, https://d3b35uem336dgh.cloudfront.net/wp-content/uploads/2018/10/do-spaces-768x327.png 768w, https://d3b35uem336dgh.cloudfront.net/wp-content/uploads/2018/10/do-spaces-1536x655.png 1536w, https://d3b35uem336dgh.cloudfront.net/wp-content/uploads/2018/10/do-spaces-2048x873.png 2048w" sizes="auto, (max-width: 2496px) 100vw, 2496px" /><br />
Storage tier-1 is used for primary storage, tier-2 is used for replica data or slave data or non-transactional data while tier-3 is used for archival or backup data or data that is rarely accessed.<br />
On a Linux system, I tend to use S3ql to mount Amazon S3 data. Because DigitalOcean (DO) has released S3-alike services for a long time, namely DO Spaces, I try to use it by making DO spaces as additional folders on Linux servers. The mounting process is done using S3ql.<br />
Below is how to create DO spaces and mount it on Linux Serve using S3ql :</p>
<ol>
<li style="list-style-type: none">
<ol>
<li>Create 1 DO spaces bucket and select the nearest region. This process just similar as if using AWS S3. If you haven&#8217;t used DO or don&#8217;t have a DO account, you can use my referral link to get an additional $ voucher : <a href="https://m.do.co/c/33ab1b84b144">DO Signup</a>.<br />
<img loading="lazy" decoding="async" class="alignnone size-full wp-image-9100" src="https://www.vavai.com/wp-content/uploads/2018/10/Screenshot-2018-10-14-19.29.38.png" alt="" width="2182" height="1274" /></li>
<li>Create a bucket name that matches its function and is easy to remember. Don&#8217;t forget to choose the appropriate option (for example activating CDN so that it is easily accessible in different geographical locations, disable listing files etc.)</li>
<li>Click on the side menu, select the <strong>API</strong> menu. In the API view, select the Spaces access keys section, create AccessKey and Secret (password) by clicking on the <strong>&#8220;Generate New Key&#8221;</strong> button</li>
<li>Write down the bucket name that has been created, Access Key and its Secret / Password</li>
<li>On the Linux server that you want to use, install the s3ql package. On Ubuntu, just run the command: :<br />
[code lang=&#8221;bash&#8221;]<br />
apt-get install s3ql<br />
[/code]</li>
<li>Run the following command to process the partition as well as mount the folder (eg mounted on the /srv/vavai folder using a bucket with the name excellent-vavai) :<br />
[code lang=&#8221;bash&#8221;]<br />
mkdir /srv/vavai<br />
mkfs.s3ql &#8211;plain &#8211;cachedir /var/cache/s3ql s3c://sgp1.digitaloceanspaces.com:443/excellent-vavai/<br />
mount.s3ql &#8211;cachedir /var/cache/s3ql s3c://excellent-vavai.sgp1.digitaloceanspaces.com:443/excellent-vavai /srv/vavai/ &#8211;allow-other &#8211;cachesize 1000000<br />
[/code]</li>
<li>When running the above command, you will usually be asked for a login back-end and back-end password. The login back-end is AccessKey while the back-end password is SecretKey. If you don&#8217;t want to repeating AccessKey and Password, we can create ~/.s3ql/authinfo2 files (for example /root/.s3ql/authinfo2 or /home/vavai/.s3ql/authinfo2 depending on the user&#8217;s home folder) with the following contents :<br />
[code lang=&#8221;bash&#8221;]<br />
storage-url: s3c://vavai-excellent.sgp1.digitaloceanspaces.com<br />
backend-login: YourAccessKey<br />
backend-password: YourSecret<br />
fs-passphrase: passphrase-to-encrypt<br />
[/code]<br />
Then run the mkfs command and mount it above with the additional parameter <strong>&#8211;authfile /root/.s3ql/authinfo</strong></li>
<li>After all these steps, check with the df -h command, the mount folder should already exist. We can do the testing the folder by creating or copying a file to the mount folder and then checking its contents. s3ql automatically performs the data encryption process, so that what appears next to the DO spaces is the encrypted files, but if it is checked in the mount folder, the result is the normal files and folders as they should<br />
<img loading="lazy" decoding="async" class="alignnone size-full wp-image-9101" src="https://www.vavai.com/wp-content/uploads/2018/10/Screenshot-2018-10-14-20.29.16.png" alt="" width="2042" height="1022" /></li>
</ol>
</li>
</ol>
<p>With the cost of DO spaces are relatively low compared to Amazon S3 or other object storage providers, we can choose DO spaces as an alternate infrequent access storage.</p>
<p>The post <a href="https://www.vavai.net/digitalocean-spaces-s3ql-and-mount-folder-on-linux-server-2/">DigitalOcean Spaces, S3QL and Mount Folder on Linux Server</a> appeared first on <a href="https://www.vavai.net">Vavai&#039;s Personal Notes</a>.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://www.vavai.net/digitalocean-spaces-s3ql-and-mount-folder-on-linux-server-2/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
	</channel>
</rss>

<!--
Performance optimized by W3 Total Cache. Learn more: https://www.boldgrid.com/w3-total-cache/?utm_source=w3tc&utm_medium=footer_comment&utm_campaign=free_plugin

Object Caching 8/87 objects using Disk
Page Caching using Disk: Enhanced 
Content Delivery Network via Amazon Web Services: CloudFront: d3b35uem336dgh.cloudfront.net
Lazy Loading (feed)
Minified using Disk
Database Caching 3/57 queries in 0.022 seconds using Disk

Served from: www.vavai.net @ 2026-10-29 07:09:39 by W3 Total Cache
-->