Importing data
The data that you use with the RedBrick AI platform can be stored in a number of places, you can upload data directly to the platform or integrate an external storage method to store your data like Amazon S3, Google Cloud Platform, or Azure Blob Storage. By using your own external storage method, you can manage the storage of your raw data, including granular level access and privacy control.
You can see the external storage method data model on the RedBrick AI model. When data is stored externally, your raw data will never be routed via RedBrick AI servers, nor will it be downloaded/duplicated (unless specifically requested for certain features, please review our Privacy Policy). The data is transferred directly from your storage method to your browser.
RedBrick AI external storage data model.

Using external storage involves two steps

  1. 1.
    Configuring your Storage Method to specify data storage locations, and authentication information.
  2. 2.
    Uploading an Items List containing specific URL's of the data from your storage method you wish to import.

Storage Methods

Currently, the options for external Storage Methods are:
You can create a storage method by clicking on the Storage Method on the left side bar of your account. On the storage method page, click on Create Storage Method.
Storage Method tab on the left sidebar
AWS S3
GCS
Azure
If your data is stored on a private S3 bucket, you will need to create a storage method of type AWS_S3.
Please visit the Configuring AWS Storage for RedBrick AI section for a detailed walkthrough on how to generate all the required parameters. A brief overview of each parameter is provided here:
  • Unique name: A unique identifier for this storage method.
  • Bucket Name: The name of your AWS S3 Bucket.
  • Region: The region code of the S3 bucket, e.g. us-east-2 (US East Ohio), ap-south-1 (Asia Pacific Mumbai). Check out the aws docs for a list of all the region code.
  • Access Key, Secret Key: The two keys that enable secure data operations. Follow along Configuring AWS Storage for RedBrick AI to generate the keys for your bucket.
Please visit the Configuring GCS Storage for RedBrick AI section for a detailed walkthrough on how to generate all the required parameters. A brief overview of each parameter is provided here:
  • Unique Name: A unique identifier for this storage method.
  • Bucket Name: The name of your GCS Bucket.
  • Service Account details: A JSON key that provides RedBrick AI with adequate permissions to access data within the bucket
Please visit the Configuring Azure Blob Storage for RedBrick AI section for a detailed walkthrough on how to generate all the required parameters. A brief overview of each parameter is provided here:
  • Unique Name: A unique identifier for this storage method.
  • Storage Account Name: The name of your Azure Blob Storage Account.
  • Azure Connection String: A string key that is linked to a particular storage account and can be used to connect to Azure storage.

Items List

The items list points the RedBrick AI platform to the data points in the data storage. This way you can selectively import data points from a storage method. The items list is a JSON file which comprises of a list of entries of the following format.
1
{
2
"items": ["<filepath_of_datapoint>"]
3
"name": "<name_of_datapoint>" // Needs to be unique
4
// Required for videos, optional for images
5
}
Copied!
For image uploads the items array will have only a single entry. For video uploads the items array has to contain the frames of the video in order.
Below are examples of single items list entries.
AWS S3, GCS and Azure Blob Storage
Local Storage
Public Cloud Storage
Say your datapoint image.png is stored inside a folder named folder inside the root-folder inside a bucket, the item list entry for that datapoint will be.
1
{
2
"items": ["root-folder/sub-folder/image.png"]
3
}
Copied!
Say your data point image.png is stored inside a folder named folder which is being hosted using an http server on http://127.0.0.1:8000/, the item list entry for that datapoint will be.
1
{
2
"url": "http://127.0.0.1:8000/folder/image.png"
3
}
Copied!
Say your datapoint is hosted at a public endpoint https://path/to/data/image.png, the item list entry for that datapoint will be.
1
{
2
"url": "https://path/to/data/image.png"
3
}
Copied!
Common Issue: don't include bucket name in the file path. When you configure your AWS S3, or GCS storage method, you will need to define the bucket name. The storage method is specific to a single bucket. Therefore, when you are creating your items list, make sure you don't include the bucket name in the path of the data point. The file paths start from the root folder inside your bucket
NOTE - Azure Blob Storage is an exception to that rule, For Azure Blob Storage the path start from the container name.

Image Items List

The items list for importing images into the RedBrick AI platform is simply a list of item list entries. Each entry will be a single datapoint on the RedBrick AI platform. The item list below will import three data points into the RedBrick AI platform.
1
[
2
{
3
"items": ["root-folder/sub-folder/image1.png"]
4
},
5
{
6
"items": ["root-folder/sub-folder/image2.png"]
7
},
8
{
9
"items": ["root-folder/sub-folder/image3.png"]
10
}
11
]
Copied!

Video Items List

The items list for importing images into the RedBrick AI platform is slightly different than images. Videos have to be parsed into frames and be imported into the RedBrick AI platform. Say you have two videos - video1, video2, that you want to import into the platform, and each video has three frames - frame1.png, frame2.png, frame3.png. The items list for this would be.
1
[
2
{
3
"name": "video-1",
4
"items": [
5
"root-folder/video-1/frame1.png",
6
"root-folder/video-1/frame2.png",
7
"root-folder/video-1/frame3.png",
8
]
9
},
10
{
11
"name": "video-2",
12
"items": [
13
"root-folder/video-2/frame1.png",
14
"root-folder/video-2/frame2.png",
15
"root-folder/video-2/frame3.png",
16
]
17
}
18
]
Copied!
Using this items list, two video data points (video1, video2) will be imported into the platform with three frames each. The frames of each video will be ordered in the same order as their appearance in the items list.
Last modified 1mo ago