Mountpoint for Amazon S3 is an open supply file shopper that makes it straightforward to your file-aware Linux purposes to attach on to Amazon Easy Storage Service (Amazon S3) buckets. Introduced earlier this 12 months as an alpha launch, it’s now usually out there and prepared for manufacturing use in your large-scale read-heavy purposes: information lakes, machine studying coaching, picture rendering, autonomous car simulation, ETL, and extra. It helps file-based workloads that carry out sequential and random reads, sequential (append solely) writes, and that don’t want full POSIX semantics.

Why Recordsdata?
Many AWS clients use the S3 APIs and the AWS SDKs to construct purposes that may record, entry, and course of the contents of an S3 bucket. Nonetheless, many purchasers have present purposes, instructions, instruments, and workflows that know how you can entry recordsdata in UNIX model: studying directories, opening & studying present recordsdata, and creating & writing new ones. These clients have requested us for an official, enterprise-ready shopper that helps performant entry to S3 at scale. After talking with these clients and asking numerous questions, we realized that efficiency and stability had been their major considerations, and that POSIX compliance was not a necessity.

Once I first wrote about Amazon S3 again in 2006 I used to be very clear that it was supposed for use as an object retailer, not as a file system. Whereas you wouldn’t need use the Mountpoint / S3 combo to retailer your Git repositories or the like, utilizing it along with instruments that may learn and write recordsdata, whereas making the most of S3’s scale and sturdiness, is smart in lots of conditions.

All About Mountpoint
Mountpoint is conceptually quite simple. You create a mount level and mount an Amazon S3 bucket (or a path inside a bucket) on the mount level, after which entry the bucket utilizing shell instructions (ls, cat, dd, discover, and so forth), library features (open, shut, learn, write, creat, opendir, and so forth) or equal instructions and features as supported within the instruments and languages that you simply already use.

Beneath the covers, the Linux Digital Filesystem (VFS) interprets these operations into calls to Mountpoint, which in turns interprets them into calls to S3: LIST, GET, PUT, and so forth. Mountpoint strives to make good use of community bandwidth, rising throughput and permitting you to cut back your compute prices by getting extra work finished in much less time.

Mountpoint can be utilized from an Amazon Elastic Compute Cloud (Amazon EC2) occasion, or inside an Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (EKS) container. It may also be put in in your present on-premises methods, with entry to S3 both straight or over an AWS Direct Join connection through AWS PrivateLink for Amazon S3.

Putting in and Utilizing Mountpoint for Amazon S3
Mountpoint is accessible in RPM format and might simply be put in on an EC2 occasion operating Amazon Linux. I merely fetch the RPM and set up it utilizing yum:

$ wget https://s3.amazonaws.com/mountpoint-s3-release/newest/x86_64/mount-s3.rpm
$ sudo yum set up ./mount-s3.rpm

For the final couple of years I’ve been often fetching pictures from a number of of the Washington State Ferry webcams and storing them in my wsdot-ferry bucket:

I acquire these pictures with a view to observe the comings and goings of the ferries, with a aim of analyzing them in some unspecified time in the future to seek out the perfect instances to experience. My aim at present is to create a film that mixes a complete day’s price of pictures into a pleasant time lapse. I begin by making a mount level and mounting the bucket:

$ mkdir wsdot-ferry
$  mount-s3 wsdot-ferry wsdot-ferry

I can traverse the mount level and examine the bucket:

$ cd wsdot-ferry
$ ls -l | head -10
whole 0
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2020_12_30
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2020_12_31
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2021_01_01
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2021_01_02
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2021_01_03
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2021_01_04
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2021_01_05
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2021_01_06
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2021_01_07
$
$  cd 2020_12_30
$ ls -l
whole 0
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 fauntleroy_holding
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 fauntleroy_way
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 lincoln
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 trenton
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 vashon_112_north
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 vashon_112_south
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 vashon_bunker_north
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 vashon_bunker_south
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 vashon_holding
$
$ cd fauntleroy_holding
$  ls -l | head -10
whole 2680
-rw-r--r-- 1 jeff jeff  19337 Feb 10  2021 17-12-01.jpg
-rw-r--r-- 1 jeff jeff  19380 Feb 10  2021 17-15-01.jpg
-rw-r--r-- 1 jeff jeff  19080 Feb 10  2021 17-18-01.jpg
-rw-r--r-- 1 jeff jeff  17700 Feb 10  2021 17-21-01.jpg
-rw-r--r-- 1 jeff jeff  17016 Feb 10  2021 17-24-01.jpg
-rw-r--r-- 1 jeff jeff  16638 Feb 10  2021 17-27-01.jpg
-rw-r--r-- 1 jeff jeff  16713 Feb 10  2021 17-30-01.jpg
-rw-r--r-- 1 jeff jeff  16647 Feb 10  2021 17-33-02.jpg
-rw-r--r-- 1 jeff jeff  16750 Feb 10  2021 17-36-01.jpg
$

I can create my animation with a single command:

$ ffmpeg -framerate 10 -pattern_type glob -i "*.jpg" ferry.gif

And right here’s what I get:

As you’ll be able to see, I used Mountpoint to entry the present picture recordsdata and to write down the newly created animation again to S3. Whereas it is a pretty easy demo, it does present how you should use your present instruments and expertise to course of objects in an S3 bucket. On condition that I’ve collected a number of million pictures through the years, with the ability to course of them with out explicitly syncing them to my native file system is a giant win.

Mountpoint for Amazon S3 Information
Listed here are a few issues to bear in mind when utilizing Mountpoint:

Pricing – There are not any new expenses for the usage of Mountpoint; you pay just for the underlying S3 operations. You can too use Mountpoint to entry requester-pays buckets.

Efficiency – Mountpoint is ready to reap the benefits of the elastic throughput supplied by S3, together with information switch at as much as 100 Gb/second between every EC2 occasion and S3.

Credentials – Mountpoint accesses your S3 buckets utilizing the AWS credentials which can be in impact whenever you mount the bucket. See the CONFIGURATION doc for extra info on credentials, bucket configuration, use of requester pays, some suggestions for the usage of S3 Object Lambda, and extra.

Operations & Semantics – Mountpoint helps fundamental file operations, and might learn recordsdata as much as 5 TB in dimension. It might probably record and browse present recordsdata, and it might probably create new ones. It can’t modify present recordsdata or delete directories, and it doesn’t assist symbolic hyperlinks or file locking (in the event you want POSIX semantics, check out Amazon FSx for Lustre). For extra details about the supported operations and their interpretation, learn the SEMANTICS doc.

Storage Lessons – You should utilize Mountpoint to entry S3 objects in all storage lessons besides S3 Glacier Versatile Retrieval, S3 Glacier Deep Archive, S3 Clever-Tiering Archive Entry Tier, and S3 Clever-Tiering Deep Archive Entry Tier.

Open Supply – Mountpoint is open supply and has a public roadmap. Your contributions are welcome; remember to learn our Contributing Pointers and our Code of Conduct first.

Hop On
As you’ll be able to see, Mountpoint is de facto cool and I’m guessing that you will discover some superior methods to place it to make use of in your purposes. Test it out and let me know what you assume!

— Jeff;