Thank you for writing this! It's crazy to think that a device we've had since forever -- the index at the back of a book -- is the key to harnessing all this information. That and the fact that you can store so much data in RAM on modern machines and just zip through it at will!
What I find so incomprehensible is that you had to do this in the first place. Making large datasets searchable is the first task, not the last, when creating a functional database. At least, that's how it was when I learned programming back when dinosaurs roamed the earth.
This is the best explanation of Search in databases that I have ever read. DOGE needs to modernize the Patent Office. My friend is an IP Paralegal and the Patent Office processes have room for big improvement.
I follow you on X as well. Thank you for explaining your work! I wish I had your talent.
DR - that is fantastic, thank you.
I love the Plain English about Google and web sites - that resonates. We need to get that to X and broadcast it.
Well done!
Thank you for writing this! It's crazy to think that a device we've had since forever -- the index at the back of a book -- is the key to harnessing all this information. That and the fact that you can store so much data in RAM on modern machines and just zip through it at will!
These (amongst others) are things we figured out at Google decades ago, it is great to see them applied here to government transparency. I hope you have plans to join with other public dbs like opensecrets or those at https://console.cloud.google.com/marketplace/browse?filter=solution-type:dataset&inv=1&invt=Abp9SA to discover more insights.
Did you build your reverse index with PostgreSQL's native full-text indexing tools, or is it entirely custom? Or a mix?
Entirely custom, and it runs entirely on the front end. There is no actual database engine or backend involved.
What I find so incomprehensible is that you had to do this in the first place. Making large datasets searchable is the first task, not the last, when creating a functional database. At least, that's how it was when I learned programming back when dinosaurs roamed the earth.
What is the role of DOGE in all this? What is the relationship between UsaSpending and DOGE and DataRepublican?
This is the best explanation of Search in databases that I have ever read. DOGE needs to modernize the Patent Office. My friend is an IP Paralegal and the Patent Office processes have room for big improvement.