Here's an idea for Flock: If you truly care about transparency, then make a directory/listing of all of your customers' transparency portals. 🖕

As it stands, I can continue aggregating the data from the portals already added to the site with some workarounds, so I will continue doing that as long as possible, or until it becomes unaffordable. So... no plans of stopping, but things will likely be changing. 🫤

If I fail to find a suitable fix, then the site will unfortunately have to move to a user-submitted format, where the only new portals that are added to the site would be from visitors submitting them.

The bandwidth I would need to purchase from a rotating proxy service would be very unaffordable as well. I would estimate near $1K/mo, if not more, with my current provider. I'm still looking into options and potential workarounds so I can continue doing automated portal discovery.

Scraping for new portals is almost certainly out of the picture due to the sheer number of requests that need to be made (millions). It would take weeks, if not months to finish a single portal discovery job with the current rate limiting.

Thanks, will get that added today.