mebkorea

5 ms·

1.
by mebkorea 5mo ago
Sure, here you go: https://lar.liverpool.gov.uk/planning/index.html?fa=search Heads up tho... it's behind AWS WAF with a JS challenge. Solving the challenge once works fine, but the WAF rate-limits the IP after ~1
2.
by mebkorea 5mo ago
Thanks! Yup,... the freemium + paid analysis route is the obvious next step from where I am. You're right that my commercial intent is muddying the message. The sell skills, not data/ consultancy angle isn't one I'd seri
3.
by mebkorea 5mo ago
All fixed. The bug was in how I was building the postcode-to-council mapping. The script was extracting prefixes from individual application addresses, and Ceredigion's database happened to have a chunk of records with HR* postcodes (d
4.
by mebkorea 5mo ago
Cheers, and things get sticky isn't lost on me, especially after some of the feedback. Seems all the feedback is converging on the same direction namely open-source the data layer, lead with public-interest, let any commercial product
5.
by mebkorea 5mo ago
Yeah, unfortunately you are not wrong about the national tradition of wanting to pull down clever-clogs... The open source angle is something I'm increasingly considering, especially after a local government IT person made a fair point
6.
by mebkorea 5mo ago
Fair points and yeah you must be sick of unrate-limited mass scraping. I run with 1.5-3 second delays from a single residential IP and back off when portals push back, but from your side I look the same as someone hammering you. On your po
7.
by mebkorea 5mo ago
Thanks. Will look into that right away!
8.
by mebkorea 5mo ago
All good points. I've been so busy with the data collection and just "irl stuff" that I haven't spoken to planners directly which is an oversight on my part — they're the obvious power users. RTPI/RICS are both
9.
by mebkorea 5mo ago
Nice thinking. Hadn't thought of DevTools MCP that way. Curl_cffi I've used for TLS fingerprinting (Edinburgh was the first one) but the discovery side I've been doing manually... open DevTools, look at the request, copy as c
10.
by mebkorea 5mo ago
Cheers, appreciate the feedback. The architect/consultant precedent angle is interesting and a couple of other commenters have already nudged me in similar directions. Tbh... you're likely right that the strongest commercial play
11.
by mebkorea 5mo ago
Updated and pushed live: planninglens.co.uk/terms. Acceptable Use clause now permits programmatic access that respects rate limits, while still protecting our derived analysis and reports. Thanks for the kick.
12.
by mebkorea 5mo ago
Thanks for the feedback. On TOS: the same answer as I gave others... the data is statutorily public, I respect rate limits. That being said, I admit it's a grey area I haven't 100% nailed down. The postcode bug is more concerning.
13.
by mebkorea 5mo ago
Thanks and yeah, some of my boundary data (for the choropleth) comes from ONS open boundary files which I think are OGL but I'd need to check the chain of derivation. On the data.gov.uk standardisation, I've seen it but last I loo
14.
by mebkorea 5mo ago
Fair points and I appreciate the feedback. Database right is real but the threshold is "substantial part". I'm literally only showing aggregates and letting people search by postcode. I'm not completely republishing coun
15.
by mebkorea 5mo ago
Honest answer... I don't fully know, zero paying customers so it's still very much a hypothesis. The two use cases I think hold up: (1) people pre-buying a house with extension potential, who otherwise guess or pay £500+ for a pla
16.
by mebkorea 5mo ago
Planning data on gov.uk is really patchy and not useful for what I want. There's planning.data.gov.uk which has some boundary/policy data but no actual decisions. The decisions only exist on council portals, which is the whole rea
17.
by mebkorea 5mo ago
Cheers! Yeah, it's honestly mental how fragmented it is. Every council is its own little island. On the shutting-off worry: the data is statutorily public. Councils are legally required to publish it, and I'm respecting rate limit
18.
by mebkorea 5mo ago
Around four months part-time. The bulk was the first 6 to 8 weeks building the three main scrapers (Idox, Northgate, Ocellaweb). After that, councils on those platforms are mostly config. The rest has been a long tail of bespoke portals, ea
19.
by mebkorea 5mo ago
Yeah, that's essentially what I'm doing. Claude handles most of the look at the portal, work out the search form, write the config loop. The actual bottleneck isn't code tbh, it's that every (snowflake) council needs lik
20.
by mebkorea 5mo ago
Thanks, honestly that means a lot! Yeah, the pre-submission review idea is interesting and I've thought about it. I have the data to surface "applications similar to yours in your ward, here's what got approved and what didn&
21.
by mebkorea 5mo ago
Fair catch and pretty embarrassing... ngl. That's a generic template clause I didn't think hard enough about at the time and it's obviously contradictory given what the site does. I'll rewrite it today. The position I wa
22.
by mebkorea 5mo ago
No, I haven't tried Browserless. So far, it has all been from a single residential IP which is probably the bigger issue with Liverpool than the WAF challenge itself. Once I have a valid session cookie I can solve the JS challenge fine
23.
by mebkorea 5mo ago
Thanks, hadn't come across them. I will have a poke around and reach out. Appreciate the pointer.
24.
by mebkorea 5mo ago
No, that's absolutely a fair follow-up and not harsh at all. It's very helpful. The "be nosey about places you used to live" use case is exactly what the postcode tool should serve (thinking about it), and right now it d
25.
by mebkorea 5mo ago
Nice! Yeah, I went the other way... deterministic scrapers per portal type because once you've worked out the search form quirks for an Idox or Northgate or Ocellaweb, it's the same shape across every council using that platform.
26.
by mebkorea 5mo ago
Hadn't thought of that tbh. Worth a go on Liverpool especially... that's the AWS WAF one I'm currently blocked on and it is doing my head in. The challenge there is volume rather than access (~80k decisions to backfill), so e
27.
by mebkorea 5mo ago
Really useful feedback, cheers. Yeah, "Mixed results" is kinda rubbish as you say. It should give you something concrete before asking for anything. I'll fix that today. Fair point on the £79 upsell at the end of a £19 report
28.
Scraping 241 UK council planning portals – 2.6M decisions so far
48 points by mebkorea 5mo ago|70 comments