I would probably try something like the modded NanoGPT benchmark, first a baseline attempt and then maybe implement the normalized transformer to see if it speeds up training by as much as they claim. But then, I'm not really an AI guy...

I have! Someone else mentioned it on my earlier post, but this isn't installable via pip (one of my requirements) and appears to run on an entire other platform.

I looked at this, too, but I'm not using a dbt project and this currently didn't have coverage for a lot of the Snowflake syntax.

Thanks! It's not as straightforward as running "sqlfluff lint" in a CLI, but nothing a little Python scripting can't solve, including dealing with potential dropped comments.

#dataBS #dataengineering

Alright, this is exciting! I was thinking yesterday that SQLFluff could be way faster using Antlr as a parser (Antlr parses in rates measured by kb/s), and it's great to see someone has already put that into production. Will check this out.

Update - I tried SQLFluff on a file with 100k lines (2.5MB) and it didn't crash, but it took over 4 hours just to lint. Too slow for my use case, so I have to find something else.

Is that a linter or an entire execution engine?

#dataengineering

Same. I spent most of a year trying a Seq2Seq LSTM to summarize transcripts. Spoke to my doctor friends and came up with a list of useful features, domain knowledge about words, etc. It never worked (skill issue, problem was too hard for me at that point in my studies) and I abandoned it.

#MLSky