How to Concatenate (cat and compose) Buckets with the gsutil Utility
cat command in the
gsutil utility directs an object’s text to standard output. For example, the following command prints the text contained in
gsutil cat gs://mybucket/a.txt
Despite its name, you can’t concatenate objects with
cat, but you can concatenate objects with
compose. That is, the following command concatenates the content of
gs://mybucket and stores the combined result to
gsutil compose gs://mybucket/a.txt gs://mybucket/b.txt gs://mybucket/c.txt
When you use
compose, keep three points in mind:
- A project can perform at most 200 compose operations per second.
composeoperation can combine a maximum of 32 entities.
- A given object can be appended to at most 1,023 times.
compose is particularly helpful if you have to upload very large files to Cloud Storage. Rather than upload an entire file from one computer, you can upload portions of the file from separate computers and use compose to combine the portions.