What HPC Infrastructure was used to generate the 2019 first picture of a black hole?

Q&A
, ,
1

There has been a lot of excitement around the first picture of a black hole (and rightfully so!) and I’ve been trying to nail down the specifics of the compute infrastructure that was used (e.g., HPC? Machines? How much memory?) There was a mention of the amount of data used (in the Terabytes, I believe) but I couldn’t find any spec for how it was processed, and where. I can imagine given all the GUI and image processing needed, maybe they did it on local machines and waited it out. However, I can also imagine there was a lot of computation needed to generate the image, which would do well on a supercomputer.

I did find this post on reddit with a link to all the papers: https://www.reddit.com/r/HPC/comments/bcndgt/imaging_a_blackhole/, and notably, the third link talks about the data pipelines.

If anyone has a hint, or knows someone out there that could answer this question, the nerdlings of AskCI want to know!

2

Hey V,

here is the link to one of the codes used on GitHub (https://github.com/achael/eht-imaging), which also includes all kinds of references, maybe something is hidden in there.

In news articles they mention that they had 5 Petabytes of Raw data (1,000 pounds), which was transported using planes to two locations, one in Germany and one in Massachusetts (https://www.inverse.com/article/54833-m87-black-hole-photo-data-storage-feat and https://motherboard.vice.com/en_us/article/597m7q/reddits-data-hoarders-are-freaking-out-over-all-that-black-hole-data). They even had to wait for summer to get some drives from the South Pole.

As far as I understand it, they needed to do a lot of simulations to train their algorithms. According to the first article above some of this preprocessing was done on GPU resources of University of Arizona.

There is also a TED talk explaining the general idea: https://www.ted.com/talks/katie_bouman_what_does_a_black_hole_look_like#t-708480

Best,
Richard

3

Thanks @rberger! I got a helpful response on Reddit too (the original post I linked) and I’ll summarize here:

  • Details of correlators are in paper II: 1000 cores with 25Gbps connectors.
  • Supercomputers are mentioned in paper III: including the following direct quote:

the simulations were performed in part on the SuperMUC cluster at the LRZ in Garching, on the LOEWE cluster in CSC in Frankfurt, and on the HazelHen cluster at the HLRS in Stuttgart.

Wow, 1,000 pounds… of data. I need a few minutes to really take that in.

4

Hi Vanessa
To elaborate on Richard’s point, one of our clusters at the University of Arizona was obtained through an NSF MRI grant and was used partly for simulating black holes, particularly Sagittarius A*, the one at the middle of the Milky Way galaxy (ours). Their simulated images are remarkably similar to the published image we saw.
I spoke recently to Junhan Kim who spent his last three Arizona winters in the balmy Antarctic summer (it was still warmer in Arizona) running the telescope and collecting data.
Chris

5

That sounds like an amazing story @Chrisreidy - there are some amazing stories in there! Maybe we could hear them some day?

6

Also just for posterity and record - there were a ton of open source projects that helped with the project, there is more discussion here on Twitter:

7

I found another good link on the TACC site about how Stampede and Jetstream contributed! https://www.tacc.utexas.edu/-/peeling-back-the-darkness-of-m87