When using S3 storage, I get unexpected results if I do not add a trailing / character at the end of my path names. What is happening and what can be done about it?
In the Unix file system, the / character is protected. It specifies a branch in the file tree, so / cannot be used in the name of a file or folder. This is not the case for S3 storage. Everything in S3 storage is stored at the top level of the bucket no matter the source’s file and folder structure. However, people have become so used to having a folder structure and how it provides an organization system, that S3 interfaces pretend to have one.
When a folder is uploaded, the files in the folder will have that folder name appended to the beginning of the file name along with the / separator. Same for if files are uploaded to existing “folders” in S3. These paths are called prefixes in S3 terms. Because / is not protected in S3, all of the / characters in a path are included with the prefix, so if you want to perform a command on a prefix in S3, you must include the trailing / because it is actually a part of the prefix name.