parallel virtualize of object-store NetCDFS with VirtualiZarr via reticulate , just I need it to be as fast when run in Python!
/2
/2
September 27, 2025 at 6:35 AM
Everybody can reply
2 likes
Or will the VirtualiZarr approach make this question moot?
January 20, 2025 at 1:10 PM
Everybody can reply
1 reposts
2) With icechunk.io + virtualizarr, we built a cloud-optimized ARCO interface to NASA’s MUR SST dataset—20+ years of daily sea surface temps, accessible in seconds for less than a cup of coffee. All from original NetCDFs. 🌊🚀
🔗 lnkd.in/gHhXVcda
🔗 lnkd.in/gHhXVcda
Icechunk
Open-source, cloud-native transactional tensor storage engine
icechunk.io
April 4, 2025 at 4:23 PM
Everybody can reply
2 likes
and go chat with other @xarray.bsky.social devs at the @earthmoverhq.bsky.social l booth in the exhibition hall!
Ask them about how VirtualiZarr works with Icechunk
icechunk.io
Ask them about how VirtualiZarr works with Icechunk
icechunk.io
Icechunk
Open-source, cloud-native transactional tensor storage engine
icechunk.io
December 10, 2024 at 4:14 PM
Everybody can reply
We’re thrilled to welcome @tegnicholas.bsky.social to the team! Tom is a prolific contributor to @xarray.bsky.social and @pangeo.io. He created Xarray Datatree and VirtualiZarr. Tom is working as a Forward Engineer, supporting our customers and the broader open-source community.
February 26, 2025 at 5:04 PM
Everybody can reply
3 quotes
10 likes
honestly, I sorta wonder if the support from tooling (VirtualiZarr, GDAL) to make legacy formats more performant in modern environments winds up being a net benefit in the end
October 11, 2024 at 1:40 AM
Everybody can reply
1 reposts
I'm at #AGU2024 this week!
If you want to talk about anything @xarray.bsky.social , Pangeo, @zarr.dev , or @cworthy.bsky.social , message me!
I have a poster on the VirtualiZarr package I wrote on Wednesday morning
agu.confex.com/agu/agu24/me...
but also several other projects I've worked on...
If you want to talk about anything @xarray.bsky.social , Pangeo, @zarr.dev , or @cworthy.bsky.social , message me!
I have a poster on the VirtualiZarr package I wrote on Wednesday morning
agu.confex.com/agu/agu24/me...
but also several other projects I've worked on...
VirtualiZarr: Create Virtual Zarr Stores Using Xarray Syntax
VirtualiZarr is a tool for creating “virtual” Zarr stores. Virtual Zarr stores ...
agu.confex.com
December 10, 2024 at 4:14 PM
Everybody can reply
5 reposts
11 likes
2/ In the pilot, we used our new open source tensor storage engine #Icechunk and #VirtualiZarr to present archival NetCDF data stored in S3 as a single analysis-ready cloud-optimized (ARCO) dataset.
March 28, 2025 at 2:03 PM
Everybody can reply
2 likes
workin' on it, getting some stuff up via VirtualiZarr, I could probably craft that qmd
October 30, 2024 at 9:58 AM
Everybody can reply
There are also another 2 posters using VirtualiZarr for cool things:
1. SWOT data via converting NASA DMR++ format (incredible work by Ayush Nag):
agu.confex.com/agu/agu24/me...
2. NOAA data dissemination (by Sean Harkins):
agu.confex.com/agu/agu24/me...
1. SWOT data via converting NASA DMR++ format (incredible work by Ayush Nag):
agu.confex.com/agu/agu24/me...
2. NOAA data dissemination (by Sean Harkins):
agu.confex.com/agu/agu24/me...
Enabling Rapid Access to NASA Earthdata: A SWOT Sea Surface Height Example
NASA Earthdata has been migrating datasets to the cloud, with recent missions, ...
agu.confex.com
December 10, 2024 at 4:14 PM
Everybody can reply
We now have a public slack channel for VirtualiZarr development and questions, feel free to join!
earthmover-community.slack.com/archives/C08...
earthmover-community.slack.com/archives/C08...
Slack
earthmover-community.slack.com
February 27, 2025 at 3:40 PM
Everybody can reply
4 reposts
5 likes
3) This demonstrates the transformative potential of the virtual approach enabled by icechunk and virtualizarr. While adopting new technology presents challenges, these innovations are necessary to make large datasets accessible and democratize access across a broader range of users and services.
April 4, 2025 at 4:23 PM
Everybody can reply
I share @rabernat.bsky.social excitement about icechunk!!! On top of delivering 100x performance, it can make impossible tasks possible.
Why am I so excited to endorse #icechunk and #virtualizarr?
bsky.app/profile/eart...
Why am I so excited to endorse #icechunk and #virtualizarr?
bsky.app/profile/eart...
April 4, 2025 at 4:23 PM
Everybody can reply
3 reposts
14 likes
See you at @foss4gna.bsky.social in Reston next week!
We’re sharing:
• eoAPI for humanitarian maps
• stac-geoparquet + stac-map
• VirtualiZarr
• an eoAPI workshop (still time to sign up!)
👋 Grab a zine and say hi
developmentseed.org/blog/2025-10...
We’re sharing:
• eoAPI for humanitarian maps
• stac-geoparquet + stac-map
• VirtualiZarr
• an eoAPI workshop (still time to sign up!)
👋 Grab a zine and say hi
developmentseed.org/blog/2025-10...
October 29, 2025 at 5:06 PM
Everybody can reply
2 likes
two ways to virtualize massive multidim arrays #RStats (with Python!, and some background xml munging)
VirtualiZarr and GDAL (these are different ways to approach the same data, not aligned yet but getting there)
{reticulate} and docker to ease the software deps
gist.github.com/mdsumner/65c...
VirtualiZarr and GDAL (these are different ways to approach the same data, not aligned yet but getting there)
{reticulate} and docker to ease the software deps
gist.github.com/mdsumner/65c...
March 27, 2025 at 3:35 AM
Everybody can reply
1 reposts
4 likes
3) This demonstrates the transformative potential of the virtual approach enabled by icechunk and virtualizarr. While adopting new technology presents challenges, innovation is necessary to make large Earth datasets accessible and to democratize access across a broader range of users and services.
April 4, 2025 at 4:36 PM
Everybody can reply
1 likes
It’s also important for understanding what problem VirtualiZarr solves.
I’ve given this explanation to many people in the past (including at @cworthy.bsky.social), so I hope that this article can serve as a useful reference the next time someone wonders what @zarr.dev actually is.
I’ve given this explanation to many people in the past (including at @cworthy.bsky.social), so I hope that this article can serve as a useful reference the next time someone wonders what @zarr.dev actually is.
April 17, 2025 at 5:50 PM
Everybody can reply
2 likes
I have a few ideas:
- create kurchunk indexes on the corpus via @tomnicholas.bsky.social’s VirtualiZarr github.com/zarr-develop...
- Use @openclimatefix.org’s hypergrib github.com/JackKelly/hy... 1/
- create kurchunk indexes on the corpus via @tomnicholas.bsky.social’s VirtualiZarr github.com/zarr-develop...
- Use @openclimatefix.org’s hypergrib github.com/JackKelly/hy... 1/
GitHub - zarr-developers/VirtualiZarr: Create virtual Zarr stores from archival data files using xarray syntax
Create virtual Zarr stores from archival data files using xarray syntax - zarr-developers/VirtualiZarr
github.com
December 7, 2024 at 8:24 PM
Everybody can reply
1 reposts
3 likes
📣 Our team is heading to the
@cloudnativegeo.org
Conference in Snowbird, Utah. We’ll be sharing our latest open-source work on Xarray, Zarr, VirtualiZarr, Icechunk, and leading a hands-on workshop, 𝑍𝑎𝑟𝑟, 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘, & 𝑋𝑎𝑟𝑟𝑎𝑦 𝑓𝑜𝑟 𝐶𝑙𝑜𝑢𝑑-𝑛𝑎𝑡𝑖𝑣𝑒 𝐺𝑒𝑜𝑠𝑝𝑎𝑡𝑖𝑎𝑙 𝐷𝑎𝑡𝑎-𝑐𝑢𝑏𝑒 𝐴𝑛𝑎𝑙𝑦𝑠𝑖𝑠.
Read more: bit.ly/4lRWNAJ
@cloudnativegeo.org
Conference in Snowbird, Utah. We’ll be sharing our latest open-source work on Xarray, Zarr, VirtualiZarr, Icechunk, and leading a hands-on workshop, 𝑍𝑎𝑟𝑟, 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘, & 𝑋𝑎𝑟𝑟𝑎𝑦 𝑓𝑜𝑟 𝐶𝑙𝑜𝑢𝑑-𝑛𝑎𝑡𝑖𝑣𝑒 𝐺𝑒𝑜𝑠𝑝𝑎𝑡𝑖𝑎𝑙 𝐷𝑎𝑡𝑎-𝑐𝑢𝑏𝑒 𝐴𝑛𝑎𝑙𝑦𝑠𝑖𝑠.
Read more: bit.ly/4lRWNAJ
April 28, 2025 at 7:33 PM
Everybody can reply
1 reposts
2 quotes
7 likes
Struggling with huge geospatial datasets in the cloud? @earthmover.io's Tom Nicolas #CNG2025 talk on VirtualiZarr + IceChunk will show you how to optimize data for cloud-native environments in just 3 lines of Python code. No more copying data
▶️ youtu.be/QBkZQ53vE6o
▶️ youtu.be/QBkZQ53vE6o
[CNG 2025] VirtualiZarr + Icechunk: Build Cloud-Optimised DataCube of Archival Files – Tom Nicholas
YouTube video by Cloud-Native Geospatial Forum
youtu.be
June 25, 2025 at 3:17 PM
Everybody can reply
🌍 IN21D: Scaling Data Discovery w Dynamic Spatial Aggregations Using STAC Metadata (@emmapaz.com)
🌍 GC23A: Choosing Earth Foundation Models: From Research Grade to Community Maintained (Wei Ji Leong)
🌍 IN31C: VirtualiZarr: Create Virtual Zarr Stores Using Xarray Syntax (Aimee, Max & Sean)
#AGU24
🌍 GC23A: Choosing Earth Foundation Models: From Research Grade to Community Maintained (Wei Ji Leong)
🌍 IN31C: VirtualiZarr: Create Virtual Zarr Stores Using Xarray Syntax (Aimee, Max & Sean)
#AGU24
December 6, 2024 at 12:45 AM
Everybody can reply
1 reposts
4 likes
Impressive overview of how VirtualiZarr and Icechunk work together when handling large numbers of files e.g. NetCDF youtu.be/rD3eeK56tJY?... @tegnicholas.bsky.social
Tom Nicholas - How to build a cloud-optimised datacube of archival files in 3 lines of xarray
YouTube video by SciPy
youtu.be
August 8, 2025 at 10:30 AM
Everybody can reply
1 likes