Joe Hamman
@jhamman.bsky.social
Climate scientist, engineer and human. Co-founder/CTO @earthmoverHQ. @xarray_dev / @pangeo_data dev. Formerly
@carbonplanorg, @NCAR_Science, and @uw_cee. He/him.
@carbonplanorg, @NCAR_Science, and @uw_cee. He/him.
Reposted by Joe Hamman
Wildfire season may be abating, but our need to understand future scenarios has not: Come join CTO
@jhamman.bsky.social Thursday morning for a workshop: Predicting cloud native wildfire risk with the Earthmover platform: m.earthmover.io/predictingwi...
@jhamman.bsky.social Thursday morning for a workshop: Predicting cloud native wildfire risk with the Earthmover platform: m.earthmover.io/predictingwi...
November 4, 2025 at 10:05 PM
Wildfire season may be abating, but our need to understand future scenarios has not: Come join CTO
@jhamman.bsky.social Thursday morning for a workshop: Predicting cloud native wildfire risk with the Earthmover platform: m.earthmover.io/predictingwi...
@jhamman.bsky.social Thursday morning for a workshop: Predicting cloud native wildfire risk with the Earthmover platform: m.earthmover.io/predictingwi...
Reposted by Joe Hamman
Meet @rabernat.bsky.social, @jhamman.bsky.social and more of the Earthmover team at our #ClimateWeekNYC events. PSA: the hands-on tech workshop Wednesday and panel on Thursday still have capacity, but are filling up fast.
September 23, 2025 at 2:00 PM
Meet @rabernat.bsky.social, @jhamman.bsky.social and more of the Earthmover team at our #ClimateWeekNYC events. PSA: the hands-on tech workshop Wednesday and panel on Thursday still have capacity, but are filling up fast.
Super excited to see #icechunk v1.0 ship today. Stable format, stable API, and ready for production. Take it for a spin and let us know how it goes. 🚀🚀🚀
Today at SciPy 2025 we released Icechunk 1.0, an open source package and specification that enables database-style transactions against petabyte-scale array datasets using only cloud object storage as infrastructure. Read about it on our blog earthmover.io/blog/icechun..., or visit earthmover.io
Icechunk 1.0: Production-Grade Cloud-Native Array Storage Is Here - Earthmover
A year ago, we made an important internal decision which set Earthmover on a new course—we decided to refactor and open source our core technology for storing array-based data in the cloud. This took ...
earthmover.io
July 10, 2025 at 7:15 PM
Super excited to see #icechunk v1.0 ship today. Stable format, stable API, and ready for production. Take it for a spin and let us know how it goes. 🚀🚀🚀
Just back from #CNG2025, I wrote up some thoughts on the traction the @zarr.dev format is gaining in the broader geospatial industry (specifically Earth Observation). I also summarize what features and tools the user and developer communities are asking for to fully leverage Zarr. 👇👇👇
1/ Our team just got back from an inspiring week at the @cloudnativegeo.org conference in Snowbird, Utah, and one thing is clear: 𝐙𝐚𝐫𝐫 𝐚𝐝𝐨𝐩𝐭𝐢𝐨𝐧 𝐢𝐬 𝐬𝐮𝐫𝐠𝐢𝐧𝐠 𝐚𝐜𝐫𝐨𝐬𝐬 𝐠𝐞𝐨𝐬𝐩𝐚𝐭𝐢𝐚𝐥.
May 7, 2025 at 3:58 PM
I'll be at the CNG conference in Snowbird next week. I wrote a short blog post about what the Earthmover team will be up to.
tldr; we'll be talking about @zarr.dev, #icechunk, @xarray.bsky.social and cloud-native data cubes.
Details in the blog post 👇
tldr; we'll be talking about @zarr.dev, #icechunk, @xarray.bsky.social and cloud-native data cubes.
Details in the blog post 👇
📣 Our team is heading to the
@cloudnativegeo.org
Conference in Snowbird, Utah. We’ll be sharing our latest open-source work on Xarray, Zarr, VirtualiZarr, Icechunk, and leading a hands-on workshop, 𝑍𝑎𝑟𝑟, 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘, & 𝑋𝑎𝑟𝑟𝑎𝑦 𝑓𝑜𝑟 𝐶𝑙𝑜𝑢𝑑-𝑛𝑎𝑡𝑖𝑣𝑒 𝐺𝑒𝑜𝑠𝑝𝑎𝑡𝑖𝑎𝑙 𝐷𝑎𝑡𝑎-𝑐𝑢𝑏𝑒 𝐴𝑛𝑎𝑙𝑦𝑠𝑖𝑠.
Read more: bit.ly/4lRWNAJ
@cloudnativegeo.org
Conference in Snowbird, Utah. We’ll be sharing our latest open-source work on Xarray, Zarr, VirtualiZarr, Icechunk, and leading a hands-on workshop, 𝑍𝑎𝑟𝑟, 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘, & 𝑋𝑎𝑟𝑟𝑎𝑦 𝑓𝑜𝑟 𝐶𝑙𝑜𝑢𝑑-𝑛𝑎𝑡𝑖𝑣𝑒 𝐺𝑒𝑜𝑠𝑝𝑎𝑡𝑖𝑎𝑙 𝐷𝑎𝑡𝑎-𝑐𝑢𝑏𝑒 𝐴𝑛𝑎𝑙𝑦𝑠𝑖𝑠.
Read more: bit.ly/4lRWNAJ
April 28, 2025 at 8:19 PM
I'll be at the CNG conference in Snowbird next week. I wrote a short blog post about what the Earthmover team will be up to.
tldr; we'll be talking about @zarr.dev, #icechunk, @xarray.bsky.social and cloud-native data cubes.
Details in the blog post 👇
tldr; we'll be talking about @zarr.dev, #icechunk, @xarray.bsky.social and cloud-native data cubes.
Details in the blog post 👇
Most people think of @zarr.dev as a "file format". With #Icechunk, we've turned Zarr into a database. @functionth.bsky.social's post shows how Icechunk can be used to solve a problem where transactional databases are often required.
1/ 🚨 New Blog Post Alert: "𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝐴𝑏𝑜𝑢𝑡 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘 𝐶𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦 𝑤𝑖𝑡ℎ 𝑎 𝐶𝑙𝑖𝑐ℎ𝑒́𝑑 𝑏𝑢𝑡 𝐼𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑣𝑒 𝐸𝑥𝑎𝑚𝑝𝑙𝑒" 🏦🔁
👉 Read it here: earthmover.io/blog/learnin...
👉 Read it here: earthmover.io/blog/learnin...
Learning about Icechunk consistency with a clichéd but instructive example - Earthmover
In this post we’ll show what can happen when more than one process write to the same Icechunk repository concurrently, and how Icechunk uses transactions and conflict resolution to guarantee consisten...
earthmover.io
April 24, 2025 at 6:40 PM
Most people think of @zarr.dev as a "file format". With #Icechunk, we've turned Zarr into a database. @functionth.bsky.social's post shows how Icechunk can be used to solve a problem where transactional databases are often required.
Reposted by Joe Hamman
1/ 🚨 New Blog Post Alert: "𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝐴𝑏𝑜𝑢𝑡 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘 𝐶𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦 𝑤𝑖𝑡ℎ 𝑎 𝐶𝑙𝑖𝑐ℎ𝑒́𝑑 𝑏𝑢𝑡 𝐼𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑣𝑒 𝐸𝑥𝑎𝑚𝑝𝑙𝑒" 🏦🔁
👉 Read it here: earthmover.io/blog/learnin...
👉 Read it here: earthmover.io/blog/learnin...
Learning about Icechunk consistency with a clichéd but instructive example - Earthmover
In this post we’ll show what can happen when more than one process write to the same Icechunk repository concurrently, and how Icechunk uses transactions and conflict resolution to guarantee consisten...
earthmover.io
April 23, 2025 at 3:30 PM
1/ 🚨 New Blog Post Alert: "𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝐴𝑏𝑜𝑢𝑡 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘 𝐶𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦 𝑤𝑖𝑡ℎ 𝑎 𝐶𝑙𝑖𝑐ℎ𝑒́𝑑 𝑏𝑢𝑡 𝐼𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑣𝑒 𝐸𝑥𝑎𝑚𝑝𝑙𝑒" 🏦🔁
👉 Read it here: earthmover.io/blog/learnin...
👉 Read it here: earthmover.io/blog/learnin...
April 21, 2025 at 6:58 PM
@zarr.dev and #icechunk are amazing but they are not magic. They are part of a thoughtfully designed cloud-native data architecture. @tegnicholas.bsky.social peels back the covers on cloud-optimized scientific data formats in our latest "Fundamentals" post 👇
1/ 💡 Our latest blog post in the fundamentals series, written by @tegnicholas.bsky.social, demystifies cloud-optimized scientific data formats!
Read more: earthmover.io/blog/fundame...
Read more: earthmover.io/blog/fundame...
Fundamentals: What is Cloud-Optimized Scientific Data?
What cloud-optimized data really means, and how Zarr and Icechunk enable fast access to massive scientific datasets in cloud object storage.
earthmover.io
April 21, 2025 at 5:12 PM
@zarr.dev and #icechunk are amazing but they are not magic. They are part of a thoughtfully designed cloud-native data architecture. @tegnicholas.bsky.social peels back the covers on cloud-optimized scientific data formats in our latest "Fundamentals" post 👇
We found similar results when we first benchmarked #icechunk. Our conclusion: doing IO with a Rust backend is much faster than Python.
👇Really exciting to see @kylebarron.dev's Obstore backend for Zarr-Python ship today.
👇Really exciting to see @kylebarron.dev's Obstore backend for Zarr-Python ship today.
With today's Zarr Python release, switch to the new obstore-based backend for **3x faster** reading from S3, Google Cloud Storage, and Azure than the default fsspec backend.
zarr.readthedocs.io/en/stable/us...
developmentseed.org/obstore/late...
zarr.readthedocs.io/en/stable/us...
developmentseed.org/obstore/late...
April 21, 2025 at 4:52 PM
We found similar results when we first benchmarked #icechunk. Our conclusion: doing IO with a Rust backend is much faster than Python.
👇Really exciting to see @kylebarron.dev's Obstore backend for Zarr-Python ship today.
👇Really exciting to see @kylebarron.dev's Obstore backend for Zarr-Python ship today.
🚀 Big launch today: introducing Flux — a game-changing way to serve geospatial data via standard APIs in seconds. Built for scale, speed, and simplicity.
1/ Today we are launching Flux, a powerful new addition to the Earthmover platform. Flux is a high-performance gateway for exploring, querying, and visualizing geospatial data via standards-compliant APIs (EDR, WMS, and OPeNDAP).
April 16, 2025 at 3:10 PM
🚀 Big launch today: introducing Flux — a game-changing way to serve geospatial data via standard APIs in seconds. Built for scale, speed, and simplicity.
Training AI models at scale from data stored in cloud object storage requires thinking carefully about both bandwidth and concurrency. In this post, @functionth.bsky.social get’s into the details of concurrent reads at scale, showing how #Icechunk and S3 can easily scale beyond 200k requests/second!
📣 Blog post alert! 𝐄𝐱𝐩𝐥𝐨𝐫𝐢𝐧𝐠 𝐈𝐜𝐞𝐜𝐡𝐮𝐧𝐤 𝐬𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲: 𝐮𝐧𝐭𝐚𝐧𝐠𝐥𝐢𝐧𝐠 𝐒𝟑'𝐬 𝐩𝐫𝐞𝐟𝐢𝐱 𝐬𝐭𝐨𝐫𝐲. This technical post by @functionth.bsky.social dives deep into the internals of how S3 shards data, showing that distributed Icechunk can easily perform 230,000 object reads/sec and beyond. earthmover.io/blog/explori...
Exploring Icechunk scalability: untangling S3's prefix story | Earthmover
We show Icechunk can scale to extremely high concurrency levels, and explain how it achieves this in modern object stores.
earthmover.io
April 9, 2025 at 7:55 PM
Training AI models at scale from data stored in cloud object storage requires thinking carefully about both bandwidth and concurrency. In this post, @functionth.bsky.social get’s into the details of concurrent reads at scale, showing how #Icechunk and S3 can easily scale beyond 200k requests/second!
Reposted by Joe Hamman
📣 Blog post alert! 𝐄𝐱𝐩𝐥𝐨𝐫𝐢𝐧𝐠 𝐈𝐜𝐞𝐜𝐡𝐮𝐧𝐤 𝐬𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲: 𝐮𝐧𝐭𝐚𝐧𝐠𝐥𝐢𝐧𝐠 𝐒𝟑'𝐬 𝐩𝐫𝐞𝐟𝐢𝐱 𝐬𝐭𝐨𝐫𝐲. This technical post by @functionth.bsky.social dives deep into the internals of how S3 shards data, showing that distributed Icechunk can easily perform 230,000 object reads/sec and beyond. earthmover.io/blog/explori...
Exploring Icechunk scalability: untangling S3's prefix story | Earthmover
We show Icechunk can scale to extremely high concurrency levels, and explain how it achieves this in modern object stores.
earthmover.io
April 9, 2025 at 3:27 PM
📣 Blog post alert! 𝐄𝐱𝐩𝐥𝐨𝐫𝐢𝐧𝐠 𝐈𝐜𝐞𝐜𝐡𝐮𝐧𝐤 𝐬𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲: 𝐮𝐧𝐭𝐚𝐧𝐠𝐥𝐢𝐧𝐠 𝐒𝟑'𝐬 𝐩𝐫𝐞𝐟𝐢𝐱 𝐬𝐭𝐨𝐫𝐲. This technical post by @functionth.bsky.social dives deep into the internals of how S3 shards data, showing that distributed Icechunk can easily perform 230,000 object reads/sec and beyond. earthmover.io/blog/explori...
We often see folks try to convince tabular data tools to perform well with multi-dimensional array data. This post by @rabernat.bsky.social explains, from first principles, why this rarely works. Its a good one! 👇👇👇
⭐ We just released the first post in our Fundamentals series. This one is called 𝐓𝐞𝐧𝐬𝐨𝐫𝐬 𝐯𝐬. 𝐓𝐚𝐛𝐥𝐞𝐬 - 𝐖𝐡𝐲 𝐭𝐚𝐛𝐮𝐥𝐚𝐫 𝐭𝐨𝐨𝐥𝐬 𝐭𝐫𝐢𝐩 𝐨𝐯𝐞𝐫 𝐠𝐫𝐢𝐝𝐝𝐞𝐝 𝐝𝐚𝐭𝐚. earthmover.io/blog/tensors...
Fundamentals: Tensors vs. Tables | Earthmover
Why tabular tools trip over gridded data.
earthmover.io
April 3, 2025 at 9:10 PM
We often see folks try to convince tabular data tools to perform well with multi-dimensional array data. This post by @rabernat.bsky.social explains, from first principles, why this rarely works. Its a good one! 👇👇👇
Super exciting to see @copernicusecmwf.bsky.social starting to adopt @zarr.dev!
What's behind ERA Explorer's speed? Meet the ZARR archive! 🚀 ZARR revolutionises data processing, enabling fast, efficient access to over 80 years of climate data.
Test out its speed 👉 era-explorer.climate.copernicus.eu?lat=50.86&ln...
#C3S @zarr.dev #beautifuldata
Test out its speed 👉 era-explorer.climate.copernicus.eu?lat=50.86&ln...
#C3S @zarr.dev #beautifuldata
April 2, 2025 at 10:52 PM
Super exciting to see @copernicusecmwf.bsky.social starting to adopt @zarr.dev!
🚨 New blog post 🚨
In it, we show off our recent work deploying #icechunk on top of #NASA's existing archives of Earth observation data. The results: 100x speed up when extracting time series from existing datasets stored as netCDF.
In it, we show off our recent work deploying #icechunk on top of #NASA's existing archives of Earth observation data. The results: 100x speed up when extracting time series from existing datasets stored as netCDF.
1/ 🚀 Solving #NASA ’s cloud data dilemma: Icechunk unlocks 100x faster access to archival data formats
We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
March 28, 2025 at 3:30 PM
This session is going to be a blast! If you are headed to CNG next month (and you should be!), consider joining us for this workshop on @xarray.bsky.social , @zarr.dev , and #icechunk. 👇👇👇
Register to attend our team's workshop at @cloudnativegeo.org! The workshop will teach attendees how to use the @zarr.dev data format model for cloud-native geospatial datacube analysis.
🗓️ Apr 30-May 2
📍Snowbird, Utah
🔗 Register here: 2025-ut.cloudnativegeo.org
🗓️ Apr 30-May 2
📍Snowbird, Utah
🔗 Register here: 2025-ut.cloudnativegeo.org
March 25, 2025 at 7:50 PM
This session is going to be a blast! If you are headed to CNG next month (and you should be!), consider joining us for this workshop on @xarray.bsky.social , @zarr.dev , and #icechunk. 👇👇👇
Reposted by Joe Hamman
1/ As a data platform company, we get a unique view of how other companies use data. In light of the recent cuts at @noaa.gov, our CEO, @rabernat.bsky.social, reached out to our customers to get their perspective on the value of NOAA’s data and services.
March 18, 2025 at 3:07 PM
1/ As a data platform company, we get a unique view of how other companies use data. In light of the recent cuts at @noaa.gov, our CEO, @rabernat.bsky.social, reached out to our customers to get their perspective on the value of NOAA’s data and services.
I'll be giving a short update tomorrow on the state of Zarr-Python following the 3.0 release back in January. Swing by the Pangeo showcase to catch the latest progress.
@zarr.dev
@zarr.dev
Join us tomorrow, March 5 at 12 PM EST for @jhamman.bsky.social's Pangeo showcase on Zarr Python 3 and beyond! Everyone's welcome to join through the Zoom link on the announcement 🤝🌎🐍
discourse.pangeo.io/t/pangeo-sho...
discourse.pangeo.io/t/pangeo-sho...
Pangeo Showcase: "Zarr Python 3 and beyond" (March 05, 2025)
Title: “Zarr Python 3 and beyond” Speaker: Joe Hamman (ORCID: 0000-0001-7479-8439) When: Wednesday, March 05, 2025 at 12 PM EST Where: Launch Meeting - Zoom Abstract: After more than a year of d...
discourse.pangeo.io
March 5, 2025 at 1:22 AM
I'll be giving a short update tomorrow on the state of Zarr-Python following the 3.0 release back in January. Swing by the Pangeo showcase to catch the latest progress.
@zarr.dev
@zarr.dev
Reposted by Joe Hamman
Denver folks — join us for a Geospatial Happy Hour on Wednesday, March 12th at 7 PM MST. Come enjoy drinks, tacos, discussing geospatial data, and meeting the Earthmover team. Register: lu.ma/o6r39p6j
Denver Geospatial Happy Hour · Luma
Come have a drink, grab a bite, and nerd out with the Earthmover team on geospatial data. Earthmover will provide food and drink tickets.
Our event will be…
lu.ma
February 27, 2025 at 9:41 PM
Denver folks — join us for a Geospatial Happy Hour on Wednesday, March 12th at 7 PM MST. Come enjoy drinks, tacos, discussing geospatial data, and meeting the Earthmover team. Register: lu.ma/o6r39p6j
Reposted by Joe Hamman
We now have a public slack channel for VirtualiZarr development and questions, feel free to join!
earthmover-community.slack.com/archives/C08...
earthmover-community.slack.com/archives/C08...
Slack
earthmover-community.slack.com
February 27, 2025 at 3:40 PM
We now have a public slack channel for VirtualiZarr development and questions, feel free to join!
earthmover-community.slack.com/archives/C08...
earthmover-community.slack.com/archives/C08...
Very, very stoked to have @tegnicholas.bsky.social join the Earthmover team!
We’re thrilled to welcome @tegnicholas.bsky.social to the team! Tom is a prolific contributor to @xarray.bsky.social and @pangeo.io. He created Xarray Datatree and VirtualiZarr. Tom is working as a Forward Engineer, supporting our customers and the broader open-source community.
February 27, 2025 at 4:06 PM
Very, very stoked to have @tegnicholas.bsky.social join the Earthmover team!
Here on Bluesky, I follow two topics: 1) software & data infrastructure and 2) cycling. I mostly post about software but today, I can share a bit about a fun cycling adventure of my own 🚵♂️.
Last week, I got to ride in the 24 Hours in the Old Pueblo - placing 4th in the Open Men Duo category:
Last week, I got to ride in the 24 Hours in the Old Pueblo - placing 4th in the Open Men Duo category:
February 24, 2025 at 5:21 PM
Here on Bluesky, I follow two topics: 1) software & data infrastructure and 2) cycling. I mostly post about software but today, I can share a bit about a fun cycling adventure of my own 🚵♂️.
Last week, I got to ride in the 24 Hours in the Old Pueblo - placing 4th in the Open Men Duo category:
Last week, I got to ride in the 24 Hours in the Old Pueblo - placing 4th in the Open Men Duo category:
Zarr-Python 3 included a bunch of performance improvements that are now landing in @xarray.bsky.social. Check out @davisvbennett.bsky.social's post on what we changed in Xarray/Zarr and how it is speeding up the "time to first byte" for large datasets 👇
1/ Check out our latest blog post earthmover.io/blog/xarray-... to learn about the dramatic improvement and performance of Xarray’s Zarr backend. We achieved improved the “time to first byte” metric, building on Zarr-Python’s new asyncio internals.
Accelerating Xarray with Zarr-Python 3 | Earthmover
We have recently dramatically improved the performance of Xarray’s Zarr backend. This post explores how we’ve improved the “time to first byte” metric, building on Zarr-Python’s new asyncio internals.
earthmover.io
February 24, 2025 at 4:56 PM
Zarr-Python 3 included a bunch of performance improvements that are now landing in @xarray.bsky.social. Check out @davisvbennett.bsky.social's post on what we changed in Xarray/Zarr and how it is speeding up the "time to first byte" for large datasets 👇