Create dataset from Google Cloud
To create a dataset from Google Cloud Storage (GCS) buckets, you will need to first launch a Workspace. Make sure to select a storage size that is at least as big as your dataset.
Once the Workspace is launched, install
Once installation is complete, restart shell:
curl https://sdk.cloud.google.com | bash
Then, you need to either login with your personal account:
exec -l $SHELL
or you can alternatively login using a service account.
gcloud auth login
Once logged in, you can download the bucket into a local folder:
Then, cd into that folder and create the Onepanel dataset as follows:
gsutil cp gs://<gs-bucket-name>/ /onepanel/input/datasets/<your-dataset-name>
cd /onepanel/input/datasets/<your-dataset-name> onepanel datasets init onepanel datasets push -b
Setting the -b option on
onepanel datasets push, will run your upload in the background. You will receive an email when the upload is complete.