8 ms·
How do you get access get all the local government meetings? Do you have a crawler that looks up every city in the country then visits each website and pull dow
by snow_mac 9mo ago
How do you get access get all the local government meetings? Do you have a crawler that looks up every city in the country then visits each website and pull down the info? A public listing site?
- nyjah 9mo agoThere isn’t any sort of standard for recording public meetings. I’ve seen everything mic less live streams with obstructed cameras to well curated flawless back and forth with great audio and transcripts. Meeting to meeting it can vary.
- whstl 9mo agoThis video that was posted here yesterday shows some details: https://youtu.be/W420BOqga_s?t=93 https://youtu.be/W420BOqga_s?t=93 Apparently there is scraping of public data + keyword matching + moderators filtering the matches. An example that he shows a bit earlier in the video comes from this page, which has an RSS feed: https://www.cityofsanbenito.com/AgendaCenter/City-Commission-2 https://www.cityofsanbenito.com/AgendaCenter/City-Commission... The video says it's open source but I can't find the source.
- stronglikedan 9mo agoTry asking. Louis is fairly responsive.
- c0brac0bra 9mo agoPerhaps something like https://www.perigon.io https://www.perigon.io?
- phildini 9mo agoHi! I've written crawlers for about a dozen municipal hosting platforms, and you can learn the bare-bones of it from our "How" page: https://civic.band/how.html https://civic.band/how.html I also gave a talk on this concept that walks through the whole process: https://www.youtube.com/watch?v=MtWzNnZvQ6w https://www.youtube.com/watch?v=MtWzNnZvQ6w The short answer is: there's no common API for any of these sites, and even the ones that do have an API are sometimes misconfigured. It's why I wrote all the scrapers by hand.