AWS Transfer Family

Learn via video courses

Overview

In today’s world, the majority of a company’s data is solely saved in a file format. Most companies save data as files, which may then be used in online, web, and mobile applications. Most of the files are transferred in real-time using protocols such as SFTP, FTP, FTPs, and AS2. The AWS Transfer family is one of the AWS services through which AWS provides managed support for file transfer to and from AWS storage services such as Amazon Simple Storage System (S3) and Amazon Elastic File System (EFS) over protocols such as SFTP, FTP, FTPs, and AS2(Applicability statement 2).

What is AWS Transfer Family?

The AWS Transfer family is used to share data from inside the AWS Storage solutions with third parties. Customers can download or upload data from their on-premise server to AWS storage solutions such as Amazon Simple Storage Solution (S3) or Amazon Elastic File System (EFS) without making any changes to their application or workflow. With AWS Transfer Family, customers don't have to manage any infrastructure since the file server provisioned by the customer will be managed by AWS.

Features of AWS Transfer Family

Identity Management

The AWS Transfer family supports multiple identity solutions. Users can authenticate using any one of the below identity services.

AWS Transfer Family Service Managed

  • Simple Key based authentication

AWS Directory Service or AWS Managed Microsoft AD

  • Microsoft AD—Customers who need password-based authentication can opt for this service.

Custom Identity Provider

  • Amazon API Gateway and AWS Lambda are used for providing custom identity solutions as per the customer’s requirements.
ActionAWS Transfer Family Service ManagedAWS Directory ServiceCustom Identity Provider
Logical Home DirectoryYesYesYes
IAM and POSIXYesYesYes
Ad hoc access structureYesNoYes
Password authenticationNoYesYes
Rate limiting built-inNoYesYes
Key-based authenticationYesNoYes

Managed Workflow for Secure File Transfer

AWS provides several prebuilt API actions, allowing clients to choose between copying, deleting, and tagging actions for their files. In addition to that, customers will also have the custom option for lambda invocation for file processing, which cannot be done by the prebuilt API.

Monitoring File Server Metrics

Amazon Cloudwatch provides the following monitoring metrics for the provisioned file server in the AWS Transfer Family. so that customers get to visualize the data transfer in and out.

NoMetricsDescription
1BytesInTotal number of bytes transferred into the AWS Storage solution
2BytesOutTotal number of bytes transferred out from the AWS Storage Solution
3FilesInTotal number of files transferred into the AWS Storage solution
4FilesOutText Total number of files transferred into the AWS Storage solution

Elastic Resource

The AWS Transfer Family File server has built-in auto-scaling capabilities. So, depending on our file transmission, the file server on the backend will dynamically scale in and scale out. According to AWS, the endpoints that we provide for the file server are designed to be available 24 x 7 for 365 days.

Native AWS Service

Once the data/files are uploaded to the AWS storage solution such as Amazon S3 or Amazon EFS,

  • Those files can be encrypted by the AWS Key Management Service.
  • Those files uploaded to S3 can be used as a source for the data lake.
  • Many AWS Services, such as AWS Translate and Amazon Athena, can be used for file processing once the file is uploaded to Amazon S3.

How AWS Transfer Family Works?

  • Initially, we need to set up a File Transfer Server with our desired protocol, such as SFTP, FTPS, or FTP. Clients can download or upload files to AWS storage services such as EFS or S3 once the file server comes online.
  • From the storage service, other processes can pick up that file, such as Data Lake.
  • From the data lake, customers can perform analytics and machine learning, or customers can directly put their data into the database for further access and processing
  • All these file transfers in and out can be achieved without changing any client’s or user’s application or workflow.
  • The file server will be managed by AWS, so customers don't have to worry about managing the infrastructure.

how-aws-transfer-family-works

Use Cases for AWS Transfer Family

The AWS Transfer Family is mainly used for the three below-mentioned use cases:

NoUsecaseDescription
1Simple File SharingIf external people from the customer’s network need to access the data over the AWS service such as Amazon S3 or Amazon EFS, They can access that data/file over common protocols such as SFTP, FTPS, or FTP. 
2Managed File TransferA dedicated IT team is responsible for operating and managing the data and file exchange securely 
3B2B IntegrationEDI (Electronic Data Interchange) data can be used with the customer's internal application integration using the AS2 protocol.

How to Get Started with the AWS Transfer Family?

Now we are going to create one SFTP file server on the console. There are eight steps involved in creating the file server.

  • Go to the AWS management console and enter the AWS Transfer Family in the search box and click the AWS Transfer Family.

    how-to-get-started-with-the-aws-transfer-family

  • In the AWS Transfer Family Console, click the Create server button.

    how-to-get-started-with-the-aws-transfer-family-2

Step - 1:
Choose Protocol Options

  • Select the SFTP protocol as mentioned in the below image. choose-protocol-options

Step - 2:
Identity Providers

  • Select the Service Managed option among the three options. identity-providers

Step - 3:
Endpoint Configuration

  • Endpoint Type: Publicly Accessible

  • Custom Hostname-None

    endpoint-configuration

Step - 4:
Domain Choice

  • Select Amazon S3

    domain-choice

Step - 5:
Create a Workflow

  • Go to the AWS Transfer Family console, click the workflow option in the left navigation pane, and click the Create workflow button. create-a-workflow

  • Enter the following description: demo-workflow

  • Click the Add Step button to add the nominal step.

    create-a-workflow-2

  • In the nominal step, select and enter the below options. Select Copy File

    copy-file

    Enter the Name: workflow-copy-step Destination Bucket: bucketest12321 (Select the bucket in your account of your choice.) Key: /

    OverwriteExisting: FALSE and click "Create workflow".

    click-create-workflow

  • Review the step and click Create step

    click-create-step

  • Now the workflow is created. workflow-is-created

Step - 6:
IAM Role Creation

  • Create one IAM role with S3 full access.
  • IAM role name: s3-full-access-role

Reference for creating an IAM role

Step - 7:
Additional Options

  • Select the ‘create a new role’ option and enter the showing workflow and IAM role which we created in the last step and click the Next button.

    additional-options

    additional-options-2

Step - 8:
Review and Create

  • Click Create Server after reviewing the details. review-and-create

  • We can see the newly created file server showing as Online. We can also add users and start transferring the files. review-and-create-2

AWS Transfer Family Pricing

There are three components involved in determining the pricing of the AWS Transfer family.

They are,

  1. Endpoints enabled for Protocol per hour
  2. Data uploaded in Gigabytes
  3. Data downloaded in Gigabytes

SFTP Pricing

NoComponentsPricing
1SFTP Time is enabled on your endpoint$0.30 per hour
2Data uploads using SFTP$0.04 per gigabyte (GB) transferred
3Data downloads using SFTP$0.04 per gigabyte (GB) transferred

FTP Pricing

NoComponentsPricing
1FTP Time is enabled on your endpoint$0.30 per hour
2Data uploads using FTP$0.04 per gigabyte (GB) transferred
3Data downloads using FTP$0.04 per gigabyte (GB) transferred

FTPS Pricing

NoComponentsPricing
1FTPS Time is enabled on your endpoint$0.30 per hour
2Data uploads using FTPS$0.04 per gigabyte (GB) transferred
3Data downloads using FTPS$0.04 per gigabyte (GB) transferred

AS2 Pricing

NoComponentsPricing
1AS2 Time is enabled on your endpoint$0.30 per hour
2Cost per message sent over using AS2*$0.01 per message
3Cost per message received over using AS2*$0.01 per message

Note:
If the message size is larger than 50 MB, $0.01 will be charged per message of 50 MB size.

Benefits of AWS Transfer Family

  • As a fully managed file transfer (MTS) service, the AWS Transfer family enables the secure transfer of files into and out of Amazon S3 and Amazon EFS.
  • AWS will operate and manage all the infrastructure necessary to maintain high availability and performance for the file server
  • Customer's files are durably stored in Amazon S3 or Amazon EFS which helps us to enable to use of those files with CRM, ERP, and web-serving applications as well as home directories and developer tools
  • We can also process the file available in S3 in our data lake with any analytics or machine learning service to extract business insights.
  • There is no upfront cost, so customers only pay for the files processed.
  • Using IAM policies, customers can provide granular file access permissions.
  • AWS also supports managed file transfer for multi-region backup and disaster recovery use cases for enterprises.

Companies Using AWS Transfer Family

  • AWS Transfer Family is used majorly in Financial service industries.

  • Companies such as Verisk, Finra, and Whole Foods Market adopted AWS Transfer Family for analytics, data migration, and application integration purposes respectively.

  • Other customers include Finra, Blutv, Discover,elula, Liberty Mutual Insurance, Pearson, Myriota, WallaNews, Ohpen, OpenGamma, Belong, SOCAR, Veeva, Zillow, ThinkCX, Kontor, Celgene, QRT and New Media.

    companies-using-aws-transfer-family

Conclusion

  • The AWS Transfer Family is a secure, scalable, reliable, and fully managed file transfer service.
  • Customers can seamlessly migrate their workflow to the service without changing any attributes or parameters in their application or internal file system.
  • A common authentication system provides secure access to all users for the exchange of important files.
  • Customers' files are stored durably in Amazon S3 or Amazon ES, which helps us to enable the use of those files with CRM, ERP, and web-serving applications as well as home directories and developer tools.
  • Customers can also process the files available in S3 as a source for data lakes with any analytics or machine learning service to extract business insights.