Create dataset from AWS S3
To create a dataset from AWS S3 buckets, you will need to first launch a Workspace. Make sure to select a storage size that is at least as big as your dataset.
Once the Workspace is launched, set your AWS credentials:
export AWS_ACCESS_KEY_ID=<key-id> export AWS_SECRET_ACCESS_KEY=<secret>
aws s3 sync s3://<s3-bucket-name>/ /onepanel/input/datasets/<your-dataset-name>
cd /onepanel/input/datasets/<your-dataset-name> onepanel datasets init onepanel datasets push -b
Setting the -b option on
onepanel push, will run upload in the background. You will receive an email when upload is complete.