Amazon S3 source
51ºÚÁϲ»´òìÈ Experience Platform provides native connectivity for cloud providers like AWS, Google Cloud Platform, and Azure. You can bring your data from these systems into Experience Platform.
Cloud storage sources can bring your own data into Experience Platform without the need to download, format, or upload. Ingested data can be formatted as XDM JSON, XDM Parquet, or delimited. Every step of the process is integrated into the Sources workflow. Experience Platform allows you to bring in data from S3 through batches.
Prerequisites prerequisites
This section outlines prerequisite steps that you must complete prior to connecting your Amazon S3 account to Experience Platform.
IP address allow list
A list of IP addresses must be added to an allow list prior to working with source connectors. Failing to add your region-specific IP addresses to your allow list may lead to errors or non-performance when using sources. See the IP address allow list page for more information.
Naming constraints for files and directories
The following is a list of constraints you must account for when naming your cloud storage file or directory.
- Directory and file component names cannot exceed 255 characters.
- Directory and file names cannot end with a forward slash (
/). If provided, it will be automatically removed. - The following reserved URL characters must be properly escaped:
! ' ( ) ; @ & = + $ , % # [ ] - The following characters are not allowed:
" \ / : | < > * ?. - Illegal URL path characters not allowed. Code points like
\uE000, while valid in NTFS filenames, are not valid Unicode characters. In addition, some ASCII or Unicode characters, like control characters (0x00 to 0x1F, \u0081, etc.), are also not allowed. For rules governing Unicode strings in HTTP/1.1 see and . - The following file names are not allowed: LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, LPT9, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, PRN, AUX, NUL, CON, CLOCK$, dot character (.), and two dot characters (…).
Configure permissions on Amazon S3
To ingest a single directory with S3, you must create an Identity and Access Management (IAM) user for Platform in the S3 console and assign permissions for the following actions:
s3:GetObjects3:GetObjectVersion
The following permissions are also required for exploring and testing connectivity:
s3:ListAllMyBucketss3:ListBuckets3:GetBucketLocation
A file path like myBucket/folder/subfolder/subsubfolder/abc.csv may lead you to only access subsubfolder/abc.csv. If you want to access the subfolder, you can specify the bucket parameter in your S3 console as myBucket and the folderPath as folder/subfolder to ensure that file exploration starts at subfolder as opposed to subsubfolder/abc.csv.
Use temporary security credentials to connect Amazon S3
You can connect Amazon S3 with temporary security credentials using the s3SessionToken. This allows you to connect Amazon S3 to Platform without having to create permanent IAM credentials with Amazon Web Services, or provide access to your Amazon S3 bucket to users in untrusted environments.
Temporary security credentials work similarly to regular, long-term access key credentials, except you can configure a shorter expiry date for your temporary credentials. Expirations can be set to a few minutes after activation or up to several hours. Temporary credentials are also not contained with the user. This means that you must request for a new set of temporary credentials, when they expire.
For steps on how to generate your temporary session token, see this .
Set up your Amazon S3 source for Experience Platform on Amazon Web Services aws
Follow the steps below to learn how you can set up your Amazon S3 account for Experience Platform on Amazon Web Services (AWS).
Prerequisites
To connect your Amazon S3 account to Experience Platform on AWS, you must have the following:
- An AWS account with access to the Amazon S3 bucket or folder that you want to connect.
- The necessary IAM permissions that allow
s3:GetObjectands3:ListBucketactions.
Access the Amazon S3 source in the UI
- Navigate to the sources catalog in the Experience Platform UI.
- Select Amazon S3 and then select New account.
- Copy the IAM Role listed under Add IAM Role to allowlist. You will later use this IAM role to set up permissions for your Amazon S3 bucket.
Set up permissions
You must configure the necessary permissions in your Amazon S3 bucket to successfully ingest data from an AWS region. The policy of the bucket that you want to access must be associated with the credentials that you are using.
Follow the steps below to update your Amazon S3 bucket:
- Log in to your account in the .
- Navigate to your Amazon S3 bucket and then select Permissions.
- Edit the bucket policy and add the following permissions:
AWS with your IAM role and the value for Resource with your Amazon S3 bucket or folder.{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AEP Get Object Related Policy Prod",
"Effect": "Allow",
"Principal": {
"AWS": "{IAM_ROLE_TO_ALLOW_LIST}"
},
"Action": "s3:Get*",
"Resource": "arn:aws:s3:::{YOUR_BUCKET_NAME}/{YOUR_FOLDER_NAME}"
},
{
"Sid": "AEP List Bucket Prod",
"Effect": "Allow",
"Principal": {
"AWS": "{IAM_ROLE_TO_ALLOW_LIST}"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::{YOUR_BUCKET_NAME}"
}
]
}
Provide your Amazon S3 connection details and validate and finalize the connection
Once you have set up your permissions, return to the sources catalog in the Experience Platform UI and navigate to the Amazon S3 source set up page. Provide the name of the Amazon S3 bucket that you want to connect to and if applicable, provide the path to the specific folder within the bucket. To verify your set up, select Test connection. If successful, you can then proceed and create a dataflow to ingest cloud storage data to Experience Platform.
Next steps
The documentation below provides information on how to connect Amazon S3 to Experience Platform using APIs or the user interface: