Nimbo
Search…
Getting started

Install Nimbo

Requires python>=3.6. Only Linux/MacOS are supported at the moment.
1
pip install nimbo
Copied!
Before continuing, we recommend reading the What is Nimbo? section (if you haven't already) for a short description of how Nimbo works. You can find the source code in our GitHub repo.
All the Nimbo commands below assume you are at the root of your project

Get your AWS credentials

I've used AWS before
I'm new to AWS
Place your AWS credentials in the file ~/.aws/credentials or generate any necessary temporary tokens as per AWS or your admin's instructions.
1
[my-aws-profile]
2
aws_access_key_id = your_aws_access_key_id
3
aws_secret_access_key = your_aws_secret_access_key
Copied!
In order to use AWS programmatically from your computer, you need an access key ID and a secret access key. If you already have these credentials on your computer (at ~/.aws/credentials ) you can skip this step. Otherwise, get your AWS credentials from the AWS dashboard, from this page. Alternatively, if you have an IAM account and you are not the administrator, you can ask your administrator for further help. You can find more information on AWS credentials here.
If you are new to AWS, make sure to run the AWS CLI configuration, within the environment in which you installed Nimbo:
1
aws configure
Copied!
This will prompt you for your access key ID and your secret access key, and write them in the appropriate file in the ~/.aws folder.
Your ~/.aws/credentials file should look something like this:
1
[my-profile]
2
aws_access_key_id = your_aws_access_key_id
3
aws_secret_access_key = your_aws_secret_access_key
Copied!
You can have multiple profiles for multiple AWS accounts. If the line [my-profile] is not present or the line is [default] instead, leave the aws_profile field in the Nimbo config file as default. Otherwise set aws_profile to your chosen AWS profile name, in this case my-profile.

Setup user permissions and instance roles

Simple setup
Advanced setup
If you are not the admin of your AWS account, please ask your admin to follow the steps in this section.
In order to create a user group and instance role that gives users in your AWS account the necessary permissions to use Nimbo, run (as root or admin):
1
nimbo admin-setup <my-admin-profile>
Copied!
This will create a user group NimboUserGroup that gives minimal EC2 access and full S3 access to users in that group. It also creates an instance role NimboFullS3AccessRole (used in the config file below), which gives your instances full access to your S3 buckets. If you are an admin and don't want to give full S3 access to a user group and/or instance role, please see the Advanced setup tab above.
You can skip the next step if you are using running jobs on Nimbo with a root AWS account, or are the sole user of your account.
To add users to the NimboUserGroup, simply run:
1
nimbo add-user <my-admin-profile> <username>
Copied!
If you are not the admin of your AWS account, please ask your admin to follow the steps in this section.
In order to create a user group and instance role that gives users in your AWS account the minimal permissions to use Nimbo, run (as root or admin):
1
nimbo admin-setup <my-admin-profile> --no-s3-access
Copied!
This will create a user group NimboUserGroup that gives minimal EC2 access and no S3 access to users in that group. Make sure to create a role/instance profile that allows the instance to access the necessary S3 buckets. For example, if you create a role called SpecialProjectRole for users in your AWS org to use with Nimbo, tell them to set role parameter in their nimbo-config.yml to SpecialProjectRole.
To add users to the newly created NimboUserGroup, simply run:
1
nimbo add-user <my-admin-profile> <username>
Copied!
Additionally, you need to make sure that the users are allowed to pass the role to the instances. For example, if you want to allow only the user Pinkman to use the SpecialProjectRole , you can add the following policy to the user:
1
{
2
"Version": "2012-10-17",
3
"Statement": [{
4
"Sid": "ExamplePassRolePolicy",
5
"Effect": "Allow",
6
"Action": "iam:PassRole",
7
"Resource": "arn:aws:iam::*:role/SpecialProjectRole",
8
"Condition": {
9
"StringEquals": {
10
"aws:username": ["Pinkman"]
11
}
12
}
13
}],
14
}
Copied!

Create the Nimbo config file

The nimbo-config.yml file contains all the information needed for running jobs with Nimbo. In the config file you can specify data, access, instance, and job details for tailoring your experience. You can create a default config file with the command:
1
nimbo generate-config
Copied!
This will create a file nimbo-config.yml in current directory.This file contains several default parameters you should customise. For more info on what each parameter does, read the Nimbo config options page.
If you completed the above steps, you can already set your own aws_profile and role values in nimbo-config.yml.

Create instance key pairs

In order to establish SSH connections between your computer and the remote instance, you need to provide the name of an EC2 Key Pair. If you already have a pem key pair file on your computer, you can skip this step.
You can automatically create and download an instance key (in the region defined in nimbo-config.yml) using:
1
nimbo mk-instance-key
Copied!
The pem file will be created in the current directory, and it is ready to use.
If you don't have permissions for this, go to your Amazon EC2 console (make sure you are in your desired region - top-right corner) to create a new pem key pair on the Key Pairs section of the left side menu. This will download a your-ec2-key-pair.pem file. Place this file in your project's folder, run chmod 400 your-ec2-key-pair.pem, and set the instance_key in the Nimbo config file to the name of your key pair.
If you have an IAM account and do not have the necessary permissions to generate a key pair file, ask your admin to generate one for you.

S3 bucket setup

If you already have an S3 bucket, make sure the S3 dataset and result paths are correctly set in your nimbo-config.yml file, and skip to the “Test your setup” section.
Otherwise, create an S3 bucket with your desired name, and update nimbo-config.yml.
1
nimbo mk-bucket <bucket-name>
Copied!
Make sure the datasets and results paths in the Nimbo config file are not within each other (e.g. datasets=data and results=data/output) and that the datasets and results paths are not outisde of the project folder. This can cause issues when using the data sync commands.

You are all done with AWS configuration! It's time to test your setup.

Test your setup

Nimbo provides a command to verify that all your credentials, access variables, and config options work. Make
1
nimbo test-access
Copied!
This will launch an instance and run a small script to verify that you have the necessary read and write permissions for EC2 and S3. If everything works, you should see a success message. Otherwise, please read the error message, verify that all the options in your config file are correct, and that you have the necessary permissions. If you or your security group/instance profile is lacking permissions, please contact your admin. You can refer to the common issues section for further troubleshooting.
If you encounter errors, it has most likely to do with incorrect credentials or permissions, not with Nimbo.

If the test-access command ran successfully, you are ready to use Nimbo! Let's move your datasets to the remote storage and

Upload your datasets to S3

If you have datasets in your local_datasets_path that you want to use in your jobs, you can upload them from your local computer to S3 using:
1
nimbo push datasets
Copied!
This will send the contents of your computer's local_datasets_path to s3_datasets_path as set in your Nimbo config file. If you want to delete some content, you can delete the files locally and then run nimbo push datasets --delete , which will delete any files that don't exist in the source folder.
At run time, Nimbo will move the data on s3_datasets_path to the instance's local_datasets_path , mimicking your local project's structure. If the datasets are correctly loaded when you run the job on your computer, it should run correctly on the instance.

Conda environments

You need to have a manually created Conda environment file to make sure the cross-platform dependencies are correctly resolved. For example, if your code uses Pytorch 1.8.0 with CUDA 10.2, it should look something like this:
1
$ cat my-env.yml
2
3
name: my-conda-env
4
channels:
5
- pytorch
6
- defaults
7
dependencies:
8
- python=3.6
9
- pip
10
- pytorch=1.8.0
11
- cudatoolkit=10.2
12
- pip:
13
- requests
Copied!
Make sure you add your Conda file name to the nimbo-config.yml.
We strongly recommend installing deep learning libraries like pytorch and tensorflow through Conda (in the top-level dependencies section) instead of pip. This will make sure the cuda/cudnn dependencies are correctly installed, as our managed images have the latest nvidia drivers installed, but may not come with the exact cuda/cudnn dependencies needed for the pip package.
If you are unsure which package version to use for Conda, run conda search <package> for a list of available packages.

Let's run some jobs

You are now ready to run your first job! Go ahead and read the Running Jobs section.
Last modified 5mo ago