1/ 🚀 Solving #NASA ’s cloud data dilemma: Icechunk unlocks 100x faster access to archival data formats
We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
March 28, 2025 at 2:03 PM
Everybody can reply
3 reposts
5 quotes
23 likes
📣 Blog post alert! 𝐄𝐱𝐩𝐥𝐨𝐫𝐢𝐧𝐠 𝐈𝐜𝐞𝐜𝐡𝐮𝐧𝐤 𝐬𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲: 𝐮𝐧𝐭𝐚𝐧𝐠𝐥𝐢𝐧𝐠 𝐒𝟑'𝐬 𝐩𝐫𝐞𝐟𝐢𝐱 𝐬𝐭𝐨𝐫𝐲. This technical post by @functionth.bsky.social dives deep into the internals of how S3 shards data, showing that distributed Icechunk can easily perform 230,000 object reads/sec and beyond. earthmover.io/blog/explori...
Exploring Icechunk scalability: untangling S3's prefix story | Earthmover
We show Icechunk can scale to extremely high concurrency levels, and explain how it achieves this in modern object stores.
earthmover.io
April 9, 2025 at 3:27 PM
Everybody can reply
4 reposts
3 quotes
5 likes
Join our webinar, Data Version Control for Arrays with #Icechunk. @rabernat.bsky.social will explain how Icechunk’s transactions, snapshots, tags, + branches can add safety & flexibility to data pipelines and workflows. Register here: bit.ly/3WVPSf2
February 11, 2025 at 4:14 PM
Everybody can reply
1 reposts
2 quotes
7 likes
Zarr v3 is also a key dependency for Earthmover's open-source project Icechunk, which I'm very excited about for its potential to radically improve scientific reproducibility.
icechunk.io
icechunk.io
Icechunk
Open-source, cloud-native transactional tensor storage engine
icechunk.io
January 9, 2025 at 5:11 PM
Everybody can reply
1 likes
I am so excited to share this new project with NASA and @developmentseed.org . By applying Icechunk and the "virtualization" approach, we can make Petabytes of archival data accessible to high performance analytics. Unlocking NASA's cloud archives will have immense downstream benefits for the world.
1/ 🚀 Solving #NASA ’s cloud data dilemma: Icechunk unlocks 100x faster access to archival data formats
We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
March 28, 2025 at 2:27 PM
Everybody can reply
12 reposts
1 quotes
34 likes
@rabernat.bsky.social I’m excited about the Icechunk integration already! 🧊
December 4, 2024 at 12:03 PM
Everybody can reply
1 likes
Not sure , a bigger problem is the new icechunk variant that you can't use without the company's Rust lib
July 29, 2025 at 10:11 AM
Everybody can reply
1 likes
🚨 New blog post 🚨
In it, we show off our recent work deploying #icechunk on top of #NASA's existing archives of Earth observation data. The results: 100x speed up when extracting time series from existing datasets stored as netCDF.
In it, we show off our recent work deploying #icechunk on top of #NASA's existing archives of Earth observation data. The results: 100x speed up when extracting time series from existing datasets stored as netCDF.
1/ 🚀 Solving #NASA ’s cloud data dilemma: Icechunk unlocks 100x faster access to archival data formats
We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
March 28, 2025 at 3:30 PM
Everybody can reply
4 reposts
1 quotes
12 likes
Icechunk: Революция в хранении транзакционных тензоров для облачных решений
https://kripta.biz/posts/7AF01EA3-6110-4936-8D4C-31857A6FDE9F
https://kripta.biz/posts/7AF01EA3-6110-4936-8D4C-31857A6FDE9F
October 20, 2025 at 10:55 AM
Everybody can reply
and go chat with other @xarray.bsky.social devs at the @earthmoverhq.bsky.social l booth in the exhibition hall!
Ask them about how VirtualiZarr works with Icechunk
icechunk.io
Ask them about how VirtualiZarr works with Icechunk
icechunk.io
Icechunk
Open-source, cloud-native transactional tensor storage engine
icechunk.io
December 10, 2024 at 4:14 PM
Everybody can reply
1/ 🚨 New Blog Post Alert: "𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝐴𝑏𝑜𝑢𝑡 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘 𝐶𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦 𝑤𝑖𝑡ℎ 𝑎 𝐶𝑙𝑖𝑐ℎ𝑒́𝑑 𝑏𝑢𝑡 𝐼𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑣𝑒 𝐸𝑥𝑎𝑚𝑝𝑙𝑒" 🏦🔁
👉 Read it here: earthmover.io/blog/learnin...
👉 Read it here: earthmover.io/blog/learnin...
Learning about Icechunk consistency with a clichéd but instructive example - Earthmover
In this post we’ll show what can happen when more than one process write to the same Icechunk repository concurrently, and how Icechunk uses transactions and conflict resolution to guarantee consisten...
earthmover.io
April 23, 2025 at 3:30 PM
Everybody can reply
1 reposts
5 quotes
5 likes
Had the idea of using Icechunk (an multi-dimensional array database) for something I would never use Icechunk for
1/ 🚨 New Blog Post Alert: "𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝐴𝑏𝑜𝑢𝑡 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘 𝐶𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦 𝑤𝑖𝑡ℎ 𝑎 𝐶𝑙𝑖𝑐ℎ𝑒́𝑑 𝑏𝑢𝑡 𝐼𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑣𝑒 𝐸𝑥𝑎𝑚𝑝𝑙𝑒" 🏦🔁
👉 Read it here: earthmover.io/blog/learnin...
👉 Read it here: earthmover.io/blog/learnin...
Learning about Icechunk consistency with a clichéd but instructive example - Earthmover
In this post we’ll show what can happen when more than one process write to the same Icechunk repository concurrently, and how Icechunk uses transactions and conflict resolution to guarantee consisten...
earthmover.io
April 23, 2025 at 3:37 PM
Everybody can reply
This session is going to be a blast! If you are headed to CNG next month (and you should be!), consider joining us for this workshop on @xarray.bsky.social , @zarr.dev , and #icechunk. 👇👇👇
Register to attend our team's workshop at @cloudnativegeo.org! The workshop will teach attendees how to use the @zarr.dev data format model for cloud-native geospatial datacube analysis.
🗓️ Apr 30-May 2
📍Snowbird, Utah
🔗 Register here: 2025-ut.cloudnativegeo.org
🗓️ Apr 30-May 2
📍Snowbird, Utah
🔗 Register here: 2025-ut.cloudnativegeo.org
March 25, 2025 at 7:50 PM
Everybody can reply
@rabernat.bsky.social talking about data storage with Icechunk using metaphors that Millenials can understand.
January 14, 2025 at 11:08 PM
Everybody can reply
5 likes
I live up northern Minnesota and while I love the idea of protected bike lanes, I'm curious how snow removal would work on them.
From what I can tell the plow would drive by on the street and push all the snow into the bike lane and then it would just sit there and harden into an unrideable icechunk
From what I can tell the plow would drive by on the street and push all the snow into the bike lane and then it would just sit there and harden into an unrideable icechunk
January 10, 2025 at 2:45 PM
Everybody can reply
New #CNG2025 talk alert!
Lindsey Nield & @deepakcherian.bsky.social from @earthmover.io present on Zarr, Icechunk & Xarray for cloud-native #geospatial data cube analysis.
Watch now: youtu.be/DSxmCWfobqQ
Lindsey Nield & @deepakcherian.bsky.social from @earthmover.io present on Zarr, Icechunk & Xarray for cloud-native #geospatial data cube analysis.
Watch now: youtu.be/DSxmCWfobqQ
[CNG 2025] Zarr, Icechunk & Xarray for Cloud-Native Geospatial Data-cube Analysis – Nield & Cherian
YouTube video by Cloud-Native Geospatial Forum
youtu.be
June 5, 2025 at 7:04 PM
Everybody can reply
1 reposts
1 likes
Come to our booth at AMS where we'll be giving out demos of Arraylake and Icechunk as well as Rubik's cube lessons!
Our team is packing our bags for #AMS2025, including stickers and Earthmover Rubik’s Cubes. Stop by booth 353 to get some swag and chat with our team!
January 9, 2025 at 10:03 PM
Everybody can reply
1 likes
Maybe icechunk zarr is for you.
June 12, 2025 at 12:37 AM
Everybody can reply
Today at SciPy 2025 we released Icechunk 1.0, an open source package and specification that enables database-style transactions against petabyte-scale array datasets using only cloud object storage as infrastructure. Read about it on our blog earthmover.io/blog/icechun..., or visit earthmover.io
Icechunk 1.0: Production-Grade Cloud-Native Array Storage Is Here - Earthmover
A year ago, we made an important internal decision which set Earthmover on a new course—we decided to refactor and open source our core technology for storing array-based data in the cloud. This took ...
earthmover.io
July 10, 2025 at 7:06 PM
Everybody can reply
3 reposts
2 quotes
11 likes
Zarr is great for big multidimensional data—but not for collaboration. Our latest post by Lindsey Nield shows how Icechunk adds Git-like transactions + version control, so teams can safely share and scale Zarr workflows.
earthmover.io/blog/multi-p...
earthmover.io/blog/multi-p...
Multi-Player Mode: Why Teams That Use Zarr Need Icechunk - Earthmover
Icechunk brings Git-like version control to Zarr. Ensure atomic updates, consistent reads, and scalable collaboration for shared datasets.
earthmover.io
August 25, 2025 at 10:45 PM
Everybody can reply
The parallels between MotherDuck’s ddx storage system and @earthmoverhq.bsky.social’s #icechunk are uncanny, let alone the mission to create cloud-native DBs.
December 17, 2024 at 6:20 PM
Everybody can reply
1 quotes
6 likes
I've worked on Icechunk almost exclusively for the last six months. I'm very proud of the result; you should check it out.
1/ 🚀 Solving #NASA ’s cloud data dilemma: Icechunk unlocks 100x faster access to archival data formats
We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
March 28, 2025 at 2:19 PM
Everybody can reply
3 likes
For climate predictors and weather analysts with #climateweekNYC FOMO: We will reprise our hands-on workshop, “Replicating official NYC climate risk predictions from ERA5 using Arraylake and Icechunk," online this Thursday, October 23 at 9AM EST: join us! m.earthmover.io/replicatingo...
October 20, 2025 at 6:31 PM
Everybody can reply
3 likes