Bringing large data into Silico¶

Use this page when a dataset or checkpoint is too large to attach in the composer. Silico keeps durable files for your organization in Cloud storage. Cloud storage appears in the Files browser beside your linked clusters, and every upload on this page lands there under the shared/ folder.

Start with where the job reads the data¶

Before you move any data, decide where the job that needs it will run. The two compute types read data from different places.

The job runs on	Where it reads data	What to do with large data
A linked cluster	The cluster's own storage	Put the data on the cluster. Use the cluster's own copy tools, or pull it from Cloud storage with a download link. Silico does not attach Cloud storage files to cluster jobs.
On-demand compute	Only the inputs declared for that job, copied from Cloud storage before the command runs	Upload the files to Cloud storage, then name them when you ask for the job. Declared inputs have size limits.

Training inputs belong on the compute that trains. For a linked cluster, copy the training set to the cluster's storage first, then tell Silico the path. Cloud storage is where you keep files that many threads and compute targets use. It is not a mount that a cluster job reads from.

Upload from your computer¶

Use the browser for files that your computer holds and can send while the tab stays open.

In the composer, open Add files, code, models and choose From your computer.
In the Upload to Cloud storage dialog, enter a folder name under Destination under shared/.
Drop files or folders into the dialog, or open Choose Files and pick Files or Folder.
Choose Start upload.

Keep the browser tab open. Closing or reloading the browser pauses the upload. To continue, open the dialog again, pick the upload under Resume an upload, and reselect the same files or folder. An upload has six days to finish.

Each file becomes visible in Cloud storage when it completes. If you cancel, the files that completed remain. Hidden files, whose names start with a period, are skipped.

Upload from a cluster or another machine¶

Use the command-line transfer when the data is on a cluster login node, a lab server, or any machine other than your computer. The transfer runs on that machine and continues when you close Silico.

The machine needs:

Python 3.9 or later. No other packages are required.
curl. The generated command uses it to fetch the transfer script.
Outbound HTTPS to your Silico address and to the storage addresses in the signed upload links. For hosted Silico, the storage addresses are Amazon S3. The machine needs no cloud credentials.
Read access to the source path.

Follow these steps:

In the composer, open Add files, code, models and choose From a cluster. The Upload to Cloud storage dialog opens on the Upload from a cluster panel.
Enter Source path on cluster, the full path to one file or one folder on that machine.
Enter Destination under shared/, then choose Generate transfer command.
Choose Copy command and run the command on the source machine within one hour. The script first lists every file in the source, and the transfer counts as started only when it submits that list. If the list is not submitted within one hour of generating the command, the command expires, and you must generate a new one.
When the terminal shows Transfer code, choose Copy code and paste the code at the prompt. The code is not part of the command. A resume asks for it again. While the transfer runs, the Copy code button stays below the progress bar in the panel. Keep the code out of scripts, chat, and shared documents. To run the transfer unattended, for example under nohup or in a batch job, save the code in a file that only you can read (chmod 600) and add --token-file <path> to the command; the script refuses a file that others can read. When standard input is not a terminal, the script reads the code from it instead of prompting, so another program can pipe it in; do not type it with echo, which records the code in your shell history, and note that the permission check applies only to --token-file. Never put the code on the command line or in an environment variable.
Keep the terminal open. Progress appears in Silico after the file list is prepared.

The transfer follows these rules:

A folder that packs into an archive of 16 GiB or less arrives as one archive named after the folder, for example checkpoints.tar. A larger folder arrives as a folder of the same name that holds archive parts of about 4 GiB each, part-00001.tar, part-00002.tar, and so on, plus a small listing file that records the size and checksum of every part. The listing file is written last, so when it is present, every part is present. Extract all the parts into one directory to rebuild the folder. A single file arrives as itself.
Hidden entries, symbolic links, and special files are skipped.
Listing a folder with hundreds of thousands of files on network storage can take a long time. Generate the command right before you run it. If the listing still does not finish within the hour, make one archive of the folder first and transfer that file instead.
A transfer can move up to 5 TiB. It has six days to finish.
Do not change the files during the transfer. The script stops if a file's size or modification time changes.
To continue after an interruption, run the same command again on the same source path and enter the same transfer code. To copy the code again, click the upload progress indicator at the bottom right of the window, or open Add files, code, models, choose From a cluster, and pick the transfer under Active remote uploads. The transfer continues from the last completed part. For a folder that arrives as archive parts, parts that already completed are not sent again, but the script reads them again to record their checksums in the listing file. Do not add, remove, or edit files under the source between runs. A changed folder needs a new transfer. A new transfer needs a different destination if parts of the old one completed, because an upload never replaces a file already in Cloud storage.
For a folder, the script holds up to 64 MiB in memory for each parallel upload and uses two parallel uploads. A single file is streamed from disk with four parallel uploads and no large buffer. Add --concurrency N (1 to 8) to the command to change either number. Only when one file inside a folder is larger than about 560 GiB does the script also use temporary disk space in the system temporary directory; set TMPDIR to a location with free space if that directory is small.
To stop a transfer, choose Cancel Upload in the panel. A canceled transfer keeps only the files that completed.

Import from a link¶

Use the import when the data is in your own cloud storage, such as an S3, GCS, or Azure bucket, and no machine you control holds a copy. You create one temporary download link per file with your own cloud tools, give the links to Silico in the thread, and Silico copies the files into Cloud storage for you. You can close the browser; the copy runs on a machine that Silico operates. Your cloud credentials stay with you: Silico only receives the links. Imports must be enabled by your operator; if they are not, Silico tells you and offers the command-line transfer instead.

Create one presigned download link per file. Make the links valid for several hours, for example aws s3 presign s3://bucket/file --expires-in 43200. Each link must point at one file.
Paste the links in the thread and name a destination folder under shared/. Silico starts the import and gives you a transfer id.
Watch progress in the thread or in the upload progress indicator at the bottom right of the window. Cancel Upload stops the import; completed files are kept.
If the links expire before the import finishes, the import pauses and Silico lists the remaining files. Create fresh links for those files and paste them in the thread; the import continues and never copies a finished file again.

Plan around these limits:

One link covers one file, and an import moves at most 5 TiB.
At most two imports run at the same time for your organization.
Anyone who has a link can download that file until the link expires. Share links only in the thread that runs the import.
An import never replaces a file already in Cloud storage. To redo an import, use a different destination folder.
The import checks your organization's Cloud storage limit before it copies anything.

Pull data from Cloud storage onto a cluster¶

When a training set is already in Cloud storage and the job runs on a linked cluster, copy it to the cluster first.

In the thread, ask Silico for a download link for the stored file. The link works for at most one hour. Silico can also run the download for you in the cluster session.
Download the file on the cluster into the experiment's storage, for example with curl:
```
curl -fL -o <file> "<link>"
```
If the file is a .tar archive from a folder upload, extract it on the cluster. If the folder arrived as archive parts, download every part, one link each, and extract them all into the same directory.
Compare the downloaded size with the size Silico reports for the stored file.

Plan around these limits:

One link covers one file. Ask for one link per file.
The link works for at most one hour. A queued job that starts later than that cannot use it. Start the download from an interactive session or from a job that starts at once.
Anyone who has the link can download the file until it expires. Do not paste links into documents or shared chats.
The cluster copy is a copy. Cloud storage keeps the original.

Declared inputs for on-demand jobs¶

On-demand jobs read only the files declared for that job. Name the files by their Cloud storage path, for example shared/datasets/eval-set/, when you ask for the job. Silico copies them into the job before your command runs.

Declared inputs may total up to 32 GiB per job, with no file above 16 GiB. Direct-delivery deployments accept shard collections without a fixed file-count cap; preparing the selection is limited to 1 MiB of reference and object metadata, and the signed download manifest to 16 MiB. The metadata budget includes repeated folder visits, so select each folder once. Some deployments use a smaller limit: 32 files, 512 MiB in total, and no file above 256 MiB. The refusal names the active limit. Files that exceed it remain safe in Cloud storage; choose compute that supports their size.

An archive that fits these limits can be declared intact. A folder that arrived as archive parts can be declared by its folder path when the whole selection fits. Silico downloads the archive or its parts and listing file before the command starts; it does not unpack them. The command needs a compatible reader, or it must extract into separate writable job storage with room for the compressed input, expanded contents, and temporary files.

Silico checks the selected files before preparing the job and creates their download links after the worktree reaches the job's storage. These links last at most one hour, and may expire sooner with the signing session. Startup and downloading must fit that window; Silico cannot refresh these links during a job. This is a bounded input route, not support for multi-terabyte on-demand jobs.

Reusing a retained volume is useful only when a completed on-demand job already holds the data. It does not attach files stored on a linked cluster.

Cloud credentials¶

Silico does not use cloud storage credentials, such as AWS access keys, to fetch data. Do not paste them into a thread. Silico drops any environment variable whose name starts with AWS_ before a job starts, so AWS keys cannot reach a job through settings or a job request. Use the upload routes on this page instead: for data in your own bucket, create presigned download links and use the import, so your credentials never leave your machine. If you administer a linked cluster, you can also copy data from your own cloud bucket to the cluster with your own tools, then tell Silico the path.

Limits at a glance¶

Limit	Value
Size per upload, transfer, or import	5 TiB
Archive size above which a folder transfer arrives as archive parts	16 GiB
Files per browser upload	10,000
Uploads and transfers in progress per person	3
Imports running at the same time per organization	2
Links per import	2,000, one file per link
Time from generating a transfer command until the script submits its file list	1 hour
Time to finish an upload or transfer	6 days
Download link lifetime	1 hour
Declared inputs per on-demand job	32 GiB total, 16 GiB per file; 1 MiB discovery metadata and 16 MiB signed manifest (some deployments: 32 files, 512 MiB total, 256 MiB per file)

See Data for attaching data in the composer and How files move through compute for what each compute type keeps after a job.