Source Code Bundles
Large files can range from 100MB all the way up to 10TB, well beyond the 5GB limit on normal files.
The 5GB limit on normal files is there primarily because that's how much you can reliably upload with a single HTTP request in a reasonable amount of time. Large files are created by assembling parts, where each part can either be uploaded or copied from existing files in any bucket belonging to the same account as the large file. And just to be clear, that means a large file can be assembled from a mix of uploaded and copied parts.
Parts of a large file can be uploaded and copied in parallel,
which can significantly reduce the time it takes to upload terabytes of
data. Each part can be anywhere from 5MB to 5GB in size, and
you can pick the size that is most convenient for your application.
For best upload performance, we recommend using the
Creating Large Files
The steps to create a large file are:
The first step is to tell B2 that you're going to create a large file
Then, you need to decide where your parts are coming from. Each part can either be uploaded, or copied from an existing file in any bucket belonging to the same account as the large file. Note that large files can be assembled from a mix of uploaded and copied parts.
You need to determine the size of each part you upload and copy. For example, if you're uploading a 100GB file, you could make each part 1GB, and perform 100 uploads, 1GB at a time. Note that parts need not all be the same size. The maximum part size is 5GB, and the minimum part size is 5MB, except for the last part in a file, which has a minimum size of one byte. We recommend a part size of 100MB, which strikes a good balance between upload throughput and the ability to upload parts in parallel.
Rather than hard coding the part size, it is a good idea to use the
The parts are numbered starting at 1, up to the number of parts needed, with a maximum of 10,000 parts for one large file.
Upload a part using
Copy a part using
Finally, once all of the parts are uploaded, you can call
Managing Large Files In Progress
Any number of large files can be in progress at once. You can
For any one unfinished large file, you can use
If you have started a large file, but don't want to finish, you
Usage Charges for Large Files
In most ways, large files are treated the same as small files. The costs for the API calls are the same.
You will be charged for storage for parts that have been uploaded or copied.
Usage is counted from the time the part is stored. When you call
Large files do not require a SHA1 checksum on the entire file but Backblaze recommends one.
If the caller knows the SHA1 of the entire large file being created,
Backblaze recommends specifying the SHA1 in the fileInfo during the call to
For each part of the large file, a SHA1 is required. When uploading a part, you must provide a SHA1 checksum, which is used to validate the data sent, and which is stored internally with that part.
When you download a large file, or a range of a large file, there is no checksum
for the entire file, so the string
Accessing Large Files
Once a large file is created, you can do anything you can do with a normal file:
Large files can range in size from 5MB to 10TB.
Each large file must consist of at least 2 parts, and all of the parts except the last one must be at least 5MB in size. The last part must contain at least one byte.