Datasets with a leading "/" in the name do not show up in all s3 file programs

Community Forums Large Data Sets
1

I have a bucket on the OSN (s3 storage). Listing files in the bucket with the boto3 library returns all the files I expect to see, but files appear to be missing if I look at the bucket with tools like rclone and Cyberduck.

Here is the output from Cyberduck:

> Examples
> examples
   foo.txt
   myTestFile1.txt
> Photoneutron
...

Not in this list are files I know are present based on boto3, like /CDMS/x/y/z.

2

The issue is that files whose names start with “/” do not show up in tools like rclone or Cyberduck! Thanks to Kevin Coakley at the OSN for figuring this out!

Here is his command using aws to list all files whose names start with “/” and this does report the missing files I was expecting to see:

% aws --profile=myprofile --endpoint=https://myendpoint.org s3 ls s3://my-data//               
                           PRE ANIMAL/
                           PRE CDMS/
                           PRE CDMSBats_testing/
                           PRE NEXUS/
                           PRE testfiles/
                           PRE z1/