Exchanging Data with Recast

Clients must provide Recast with data updates to support model refreshes, and may also (optionally) receive model output data back from Recast. Recast supports many options for this data exchange, including both Recast-managed and client-managed data repositories, as well as third-party services. All of the currently supported options are listed below, as well as details of the setup process and requirements for each, particularly in regard to security and authentication.

Recast is continually implementing new data exchange options based on client needs, so if you don’t see an option below that works for your organization, please inquire. However, implementing new options is likely to extend onboarding timelines, so utilizing one of the options listed below whenever possible is encouraged.

The table of contents below provides a list of the currently supported data repositories with links to quickly access more information about each option.

1 Recast-Managed Data Repositories
- 1.1 AWS S3
- 1.2 SFTP
2 Client-Managed Data Repositories
- 2.1 AWS S3
- 2.2 SFTP
- 2.3 Google Sheets
- 2.4 Google Cloud Storage
- 2.5 Google BigQuery
- 2.6 Google Drive
- 2.7 Snowflake
- 2.8 Postgres Databases (including Redshift)
3 Third-Party Data Repositories

Recast-Managed Data Repositories

AWS S3

Clients may transfer data to and from an AWS S3 bucket in Recast’s AWS account.

This option is often preferred by clients who have their own AWS account (but don’t want to manage the S3 bucket used for data exchange with Recast) or who are using a third-party service which offers an “S3 connector”.

Two authentication methods are supported, with the details of each covered below.

IAM Role (strongly preferred)

Recast can create an IAM role which may be assumed from the client’s AWS account or the AWS account of a third-party service being utilized by the client. Once the role is assumed, access to Recast’s S3 bucket will be available.

In order to create the IAM role, Recast needs to know the following information:

Required: The AWS account number which the role will be assumed from.
Optional: The ARN of the client’s IAM user or role which will be assuming Recast’s IAM role. If not provided, Recast will allow any IAM user or role in the client’s AWS account to assume Recast’s IAM role.
Optional: The “external ID” included in requests made by the client’s IAM role to assume Recast’s IAM role. The use of an external ID is an additional security measure most appropriate when utilizing a third-party service to connect to Recast’s S3 bucket. The third-party service will provide this external ID (generally when configuring the S3 connector). Recast strongly discourages the use of any third party service which does not support an external ID for S3 connectors. For connections from client-managed AWS accounts, an external ID is less critical but is also supported.

Once Recast completes setup of the IAM role, Recast will provide the IAM role’s ARN to the client.

The following is a summary of the setup information to be exchanged between Recast and the client.

Client provides to Recast:

The account number of the client’s AWS account which Recast’s IAM role will be assumed from. (required)
The ARN of the client’s IAM user or role which will will be assuming Recast’s IAM role. (optional)
The external ID included in requests made by the client’s IAM role to assume Recast’s IAM role. (optional)

Recast provides to client:

The ARN of Recast’s IAM role which the client will be assuming.
The bucket name of Recast’s S3 bucket.
The path in the S3 bucket where the client will write and read data.

IAM User (supported but strongly discouraged)

Recast can create an IAM user with an access key which may be used to access Recast’s S3 bucket. The use of an IAM user provides great flexibility in the methods used to access Recast’s S3 bucket for clients who do not have their own AWS account, including programmatic access via the AWS API, command-line access via the AWS CLI, and third-party “S3 connectors”.

Although this option is supported, and even sometimes necessary given some current client constraints, Recast strongly discourages its use because access via an IAM user is more difficult to secure than access via an IAM role. Instead, Recast recommends the use of an IAM role (for clients with their own AWS account) or SFTP (for clients without their own AWS account) whenever possible.

Additionally, Recast’s security policies require IAM user access keys to be rotated periodically, which necessitates ongoing maintenance and may also disrupt scheduled data exchanges and model refreshes if the key rotation is not handled in a timely manner.

Recast does not require any information from the client in order to set up an IAM user for the client.

Once Recast completes setup of the IAM user, Recast will provide the IAM user’s access key and the S3 bucket name and path to the client.

This information must be communicated securely, through a service such as onetimesecret.com, or some other secure messaging channel.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

The Access Key ID and Secret Access Key of Recast’s IAM user which the client will use to access Recast’s S3 bucket.
The bucket name of Recast’s S3 bucket.
The path in the S3 bucket where the client will write and read data.

SFTP

Clients may transfer data to and from an SFTP server managed by Recast, with an SFTP user set up by Recast.

This option is often preferred by clients who do not have their own AWS account, who have an established pattern of utilizing SFTP for data transfers, or who are using a third-party service which offers an “SFTP connector”.

Two authentication methods are supported, with the details of each covered below.

Key-Based Authentication (strongly preferred)

Recast supports and recommends key-based authentication for SFTP.

In order to set up an SFTP user for the client using key-based authentication, the client must create an SSH key pair and provide Recast with its public key. (The client should not share the private key, and should store the key pair securely.)

After Recast receives the public key from the client and completes setup of the SFTP user, Recast will provide the client with the information necessary to access the SFTP server, including hostname (sftp.getrecast.com), port (22), and username.

The following is a summary of the setup information to be exchanged between Recast and the client.

Client provides to Recast:

The public key from the client-generated SSH key pair.

Recast provides to client:

The SFTP host name.
The SFTP port.
The SFTP username.

Password Authentication (supported but strongly discouraged)

Recast supports, but does not recommend, password authentication for SFTP.

Although this option is supported, Recast strongly discourages its use because password authentication is less secure than key-based authentication.

Additionally, Recast’s security policies require SFTP passwords to be rotated periodically, which necessitates ongoing maintenance and may also disrupt scheduled data exchanges and model refreshes if the key rotation is not handled in a timely manner.

Recast does not require any information from the client in order to set up password authentication.

Once setup of the SFTP user is complete, Recast will provide the client with the information necessary to access the SFTP server, including hostname (sftp.getrecast.com), port (22), username, and password.

This information must be communicated securely, through a service such as onetimesecret.com, or some other secure messaging channel.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

The SFTP host name.
The SFTP port.
The SFTP username.
The SFTP password.

Client-Managed Data Repositories

AWS S3

Recast may transfer data to and from an AWS S3 bucket owned by the client. Recast will access the client’s S3 bucket from Recast’s AWS account.

This option is often preferred by clients who have their own AWS account and want to manage the S3 bucket that is used for data exchange with Recast.

Two authentication methods are supported, with the details of each covered below.

IAM Role (preferred)

The client will create an IAM role in the client’s AWS account, which Recast will assume from Recast’s AWS account.

The IAM role’s trust policy must allow Recast to assume the role. The client may trust any IAM user or role in Recast’s account by specifying the following trust policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Resource": "arn:aws:iam::878987988123:root"
        }
    ]
}

If the client prefers a more restrictive trust policy, limiting access to a single, specific IAM role in Recast’s AWS account, the following trust policy is recommended.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Resource": "arn:aws:iam::878987988123:role/recast_access_client_s3_bucket_[client_id]"
        }
    ]
}

The placeholder text [client_id] will be a unique client identifier provided by Recast.

The client’s IAM role’s permissions must allow certain S3 actions in order for Recast to read from and write to the client’s S3 bucket. The following policy allows the S3 actions which are necessary now, as well as any that are likely to be needed in the future.

{
    "Statement": [
        {
            "Action": [
                "s3:*"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::[client_bucket_name]",
                "arn:aws:s3:::[client_bucket_name]/*"
            ]
            "Sid": "S3Access"
        },
    ],
    "Version": "2012-10-17"
}

The placeholder text [client_bucket_name] must be replaced with the actual name of the client’s S3 bucket.

Recast recommends that the client create a dedicated S3 bucket used solely for the purpose of exchanging data with Recast.

The policy above would allow all S3 actions across the entire S3 bucket, and should therefore only be used if the S3 bucket is for exclusive use by Recast. Should the client chose to use a non-dedicated bucket, it is up to the client to implement a policy which restricts Recast’s access to the bucket appropriately.

Should the client wish to take a more restrictive “principle of least privilege” approach in regard to the allowed actions, the following actions may be specified, although this more restrictive actions list may require future updates as requirements change.

"Action": [
    "s3:ListObjectsV2",
    "s3:ListBucket",
    "s3:GetObjectAttributes",
    "s3:GetBucketLocation",
    "s3:*ObjectTagging",
    "s3:*Object",
    "s3:*Acl"
]

Once the client completes setup of the IAM role, the client will provide the IAM role’s ARN to Recast.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

The ARN of the IAM role in Recast’s account which will assume the IAM role in the client’s account. (optional)

Client provides Recast:

The ARN of the IAM role in the client’s account which Recast will assume to access the client’s S3 bucket.
The bucket name of the client’s S3 bucket.

IAM User (supported)

The client will create an IAM user in the client’s AWS account, which Recast will authenticate to with the user’s access key which the client must also create.

The client’s IAM user’s permissions must allow certain S3 actions in order for Recast to read from and write to the client’s S3 bucket. The following policy allows the S3 actions which are necessary now, as well as any that are likely to be needed in the future.

{
    "Statement": [
        {
            "Action": [
                "s3:*"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::[client_bucket_name]",
                "arn:aws:s3:::[client_bucket_name]/*"
            ]
            "Sid": "S3Access"
        },
    ],
    "Version": "2012-10-17"
}

The placeholder text [client_bucket_name] must be replaced with the actual name of the client’s S3 bucket.

Recast recommends that the client create a dedicated S3 bucket used solely for the purpose of exchanging data with Recast.

The policy above would allow all S3 actions across the entire S3 bucket, and should therefore only be used if the S3 bucket is for exclusive use by Recast. Should the client chose to use a non-dedicated bucket, it is up to the client to implement a policy which restricts Recast’s access to the bucket appropriately.

Should the client wish to take a more restrictive “principle of least privilege” approach in regard to the allowed actions, the following actions may be specified, although this more restrictive actions list may require future updates as requirements change.

"Action": [
    "s3:ListObjectsV2",
    "s3:ListBucket",
    "s3:GetObjectAttributes",
    "s3:GetBucketLocation",
    "s3:*ObjectTagging",
    "s3:*Object",
    "s3:*Acl"
]

Once the client completes setup of the IAM user, the client will provide the IAM user’s access key and the S3 bucket name to Recast.

This information must be communicated securely, through a service such as onetimesecret.com, or some other secure messaging channel.

The following is a summary of the setup information to be exchanged between Recast and the client.

Client provides Recast:

The Access Key ID and Secret Access Key of the client’s IAM user which Recast will use to access the Client’s S3 bucket.
The bucket name of the Client’s S3 bucket.

SFTP

Recast may transfer data to and from an SFTP server managed by the client, with an SFTP user set up by the client.

This option is often preferred by clients who have an their own SFTP server and an established practice of exchanging data via SFTP.

Two authentication methods are supported, with the details of each covered below.

Key-Based Authentication (strongly preferred)

Recast supports and recommends key-based authentication for SFTP.

To allow the client to set up an SFTP user for Recast using key-based authentication, Recast will create an SSH key pair and provide the client with its public key. This key pair will be used exclusively for accessing the client’s SFTP server.

After the client receives the public key from Recast and completes setup of the SFTP user, the client will provide Recast with the information necessary to access the SFTP server, including hostname, port, and username.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

The public key from the Recast-generated SSH key pair.

Client provides to recast:

The SFTP host name.
The SFTP port.
The SFTP username.

Password Authentication (supported but strongly discouraged)

Recast supports, but does not recommend, password authentication for SFTP.

Although this option is supported, Recast strongly discourages its use because password authentication is less secure than key-based authentication.

Additionally, Recast’s security policies require SFTP passwords to be rotated periodically, which necessitates ongoing maintenance and may also disrupt scheduled data exchanges and model refreshes if the key rotation is not handled in a timely manner.

The client should not require any information from Recast in order to set up password authentication. The client should randomly generate a strong password.

Once setup of the SFTP user is complete, the client will provide Recast with the information necessary to access the SFTP server, including hostname, port, username, and password.

This information must be communicated securely, through a service such as onetimesecret.com, or some other secure messaging channel.

The following is a summary of the setup information to be exchanged between Recast and the client.

Client provides to Recast:

The SFTP host name.
The SFTP port.
The SFTP username.
The SFTP password.

Google Sheets

Recast may transfer data to and from Google Sheets owned by the client.

This option is often preferred by clients who are already maintaining data in Google Sheets, or who are not prepared to utilize other options such as AWS S3 or SFTP.

The client will give Recast’s Google service account permissions to the client’s sheets.

After sharing the sheets, the client must provide Recast with links (URLs) to the sheets.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

Recast’s service account (servers@ambient-hulling-282015.iam.gserviceaccount.com).

Client provides to Recast:

Link (URL) to all sheets.

Google Cloud Storage

Recast may transfer data to and from a Google Cloud Storage (GCS) bucket owned by the client.

Two authentication methods are supported, with the details of each covered below.

Service Account Authentication

The client will give Recast’s Google service account permissions to the client’s GCS bucket.

Once access to the bucket has been configured, the client will provide Recast with the name of the bucket.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

Recast’s service account (servers@ambient-hulling-282015.iam.gserviceaccount.com).

Client provides to Recast:

The bucket name of the Client’s GCS bucket.

Key-Based Authentication

Recast will provide the client with a public key.

The client will configure their GCS bucket to allow access to the bucket, authenticated by the public key.

Once access to the bucket has been configured, the client will provide Recast with the name of the bucket.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

Recast’s public key.

Client provides to Recast:

The bucket name of the Client’s GCS bucket.

Google BigQuery

Recast may transfer data to and from a Google BigQuery warehouse owned by the client.

This option is often preferred by clients who are already maintaining data in Google BigQuery.

The client will grant the appropriate BigQuery roles to a Recast service account.

Once the service account roles have been configured, the client will provide Recast with the Google BigQuery project ID and dataset ID.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

Recast’s service account (servers@ambient-hulling-282015.iam.gserviceaccount.com).

Client provides to Recast:

The BigQuery project ID.
The BigQuery dataset ID.

Google Drive

Recast may transfer data to and from a Google Drive folder owned by the client.

This option is often preferred by clients who are already using Google Drive, and who are not prepared to utilize other options such as AWS S3 or SFTP.

The client will give Recast’s Google service account permissions to the client’s Google Drive folder.

After sharing the sheets, the client must provide Recast with a link (URL) to the folder.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

Recast’s service account (servers@ambient-hulling-282015.iam.gserviceaccount.com).

Client provides to Recast:

Link (URL) to the Google Drive folder.

Snowflake

Recast may transfer data to and from a Snowflake database owned by the client.

This option is often preferred by clients who are already maintaining data in Snowflake.

Recast does not have a Snowflake account so this option is only available for clients who have their own Snowflake account. Since Recast does not have a Snowflake account, the client must set up a reader account for Recast, and cannot utilize Secure Data Sharing.

Snowflake now requires key-pair authentication for reader accounts, and no longer supports password authentication.

Recast will provide the client with the public key from a key pair generated solely for this use.

The client will create a reader account for Recast, and then provide Recast with the reader account username and the details of the Snowflake data source.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

The public key from the Recast-generated key pair.

Client provides to Recast:

The Snowflake username.
The Snowflake hostname.
The Snowflake database.
The Snowflake schema.
The Snowflake warehouse.
The Snowflake role.

Postgres Databases (including Redshift)

Recast may transfer data to and from a Postgres database owned by the client.

This option is often preferred by clients who are already maintaining data in a Postgres database.

Recast supports username/password authentication for Postgres databases.

The client should not require any information from Recast in order to set up a database user with password authentication. The client should randomly generate a strong password.

Once setup of the database user is complete, the client will provide Recast with the information necessary to access the database, including hostname, port, username, password, and database.

This information must be communicated securely, through a service such as onetimesecret.com, or some other secure messaging channel.

The following is a summary of the setup information to be exchanged between Recast and the client.

Client provides to Recast:

The database host name.
The database port.
The database username.
The database password.
The database name.

Third-Party Data Repositories

Recast also supports data exchange with third-party services, including Fivetran, Census, and Adverity. Such support is generally implemented via “connectors”, such as S3 connectors and SFTP connectors, in which case the information on those access methods above generally applies, with some specific requirements depending on the third party.

Fivetran - Supports AWS IAM roles via an S3 connector. The AWS account number will be Fivetran’s AWS account, not the client’s AWS account. An external ID is required, and provided by Fivetran, unique to each client. Also supports SFTP via an SFTP connector, although each file requires setup of a separate SFTP connector, so using an S3 connector is recommended.
Census - Supports AWS IAM roles via an S3 connector. The AWS account number will be Census’s AWS account, not the client’s AWS account. An external ID is required, and provided by Census, unique to each client.
Adverity - Supports AWS IAM roles and IAM users as S3 sources and destinations.

Although Recast generally prefers using an IAM role over an IAM user, Adverity does not support an external ID in its IAM role configuration, which Recast views as a significant security vulnerability for third-party services. Therefore Recast recommends, in this specific case, the use of an IAM user. Should the client prefer to use an IAM role to avoid the need to rotate the IAM user access key, the AWS account number will be Adverity’s AWS account, not the client’s AWS account.

Other third-party services are likely to also support access to Recast’s S3 bucket, and Recast is happy to assist with implementing access, as requested.