7 ms·

I made a simple geolocation service

340 points by maxko 6y ago

crispyporkbites 6y ago
Are you charging for this?
- maxko 6y ago
  The code is available for free and you can host it on your own Cloudflare account. Cloudflare provides a free plan (up to 100k daily requests across all your Workers), and paid plan ($0.50 per 1m requests)
PaywallBuster 6y ago
tl;dr uses Cloudflares workers as API endpoint and returns Country name based on header in the CF request.
bellwether 6y ago
Love the ingenuity and thank you for the performance comparison on Cloudflare Workers vs AWS Lambda. I personally wouldn’t consider what you built a Geolocation service, but glad it solved your use case!
- maxko 6y ago
  Thanks!
speedgoose 6y ago
Well that's a bit more than 2 requests per second on average. Sure you may have some bursts once in a while, but nothing a 3€ VM can't manage IMHO.
- Nextgrid 6y ago
  I always find it funny when people brag about cost savings on cloud services while overlooking the insane premium they're paying by using the "cloud" to begin with. It's kinda like a construction firm using a supercar to haul materials and then bragging about their efforts to make cosmetic repairs cost-effective... maybe you could just not use an overpriced car to begin with and then damage won't be an issue?
  - harpratap 6y ago
    Am I missing something? Isn't a $3 VM referring to a VM service in a cloud somewhere? If you try to do the same thing without cloud you'll have to end up renting space in a datacenter, install your own servers, connect to an ISP, power bills, etc etc. Cloud definitely seems cheaper in comparison
    - progval 6y ago
      GP and you aren't using the same definition of cloud. By cloud, they mean infrastuctures like AWS/GCP/Azure, not simple VM hosting (called VPS outside "clouds", and compute nodes in "clouds")
      - harpratap 6y ago
        What's the definition of "cloud" then which excludes VPS from the umbrella of services offered in cloud? Not really sure how a VPS is any different from a Google Compute Engine
        XCSme 6y ago
        I think the main difference is the pricing structure. Usually "cloud" refers to the auto-scaling servers with the pay-as-you go pricing, while VPSs are usually manually instantiated and have a fixed monthly cost. Also, sometimes with VPSs you get more control over the server's settings than with "cloud" services. So "cloud" is like renting a dynamically changing number of VPSs each month and which are usually more costly if your consumption is not spiky (eg. when a cheap VPS rented 24/7 is enough).
  - dmortin 6y ago
    The main advantage of the cloud for me that it provides PaaS services. You don't have to administer a VM, etc, you just use the services and it's someone else's job to install security updates and stuff.
    - 6y ago
- csunbird 6y ago
  A Pi should be able to handle this too.
  - quickthrower2 6y ago
    A Pi is basically yesteryear’s beefy rack server.
- jbverschoor 6y ago
  The only thing I like about lambda, is that your code could easily be deployed to different geo locations
  - akx 6y ago
    You can also easily spin up a VM in a different geographic location.
    - luckylion 6y ago
      That's true, but likely not as close to the customer as CF Workers on edge. And certainly not as easy, as CF does all that automatically.
    - eat_veggies 6y ago
      If you spun up a VM in 200 locations like Cloudflare, that $3 VPS suddenly becomes $600
- mikeyjk 6y ago
  A lambda is potentially cheaper if there are long periods with no execution though?
pereiratr 6y ago
Nice! Have you tried AWS Lambda@Edge?
- pereiratr 6y ago
  and it looks like similar geolocation info "cloudfront-viewer-country" is available from AWS as well. So similar trick maybe possible with AWS https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-examples.html#lambda-examples-redirect-based-on-country https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
wingi 6y ago
I had a geo-location service running in google app engine. for free. After pimping caching the daily traffic of 400.000 requests/day was in the free-tier. https://www.united-coders.com/christian-harms/detailed-performance-optimization-steps-on-webservice/ https://www.united-coders.com/christian-harms/detailed-perfo...
jasonlingx 6y ago
This one I made couple years ago (and haven’t checked in years) is still running for free thanks to Heroku and Cloudflare: https://github.com/jlxw/geoip https://github.com/jlxw/geoip
whatl3y 6y ago
So this finds the country based on some cloudflare-populated info of an incoming request (which sounds like it solved OP’s problem which is good), but if you want to use MaxMind’s database to find the country of any public IP I built the following a few months back: https://github.com/Risk3sixty-Labs/geoapi https://github.com/Risk3sixty-Labs/geoapi
EE84M3i 6y ago
What are the runtime limitations that prevent the maxmind database from working inside cloudflare workers?
- achairapart 6y ago
  I was wondering the same.
grizzles 6y ago
How does MaxMind prevent someone from releasing an open source version of their database? If you are about to answer "Copyright", remember - you can't Copyright facts. This has been upheld in the Courts system many times.
- itsjloh 6y ago
  MaxMind have two versions of their database, the free to use version called GeoLite and the paid version (whos name I'm not aware of). I realise this doesn't directly answer your question but I guess one reason might be that they already provide a free version so releasing the paid version would just get you in strife?
- deleted 6y ago
  [deleted]
- mike_d 6y ago
  Facts cannot be copyrighted, but a unique compilation of facts can be. This became settled law many decades ago with sports almanacs. Furthermore, Maxmind adds non-factual entries to the database so they can identify it uniquely as their work. Map makers do something similar: https://en.wikipedia.org/wiki/Trap_street https://en.wikipedia.org/wiki/Trap_street
- renewiltord 6y ago
  You can just stick this in a license agreement, surely.
- Mindwipe 6y ago
  Maxmind's database is updated very regularly and includes a lot of things like risk scoring. Those are absolutely protected.
- redis_mlc 6y ago
  > If you are about to answer "Copyright", remember - you can't Copyright facts. Nope, US law was changed over a decade ago.
itsjloh 6y ago
I (like many others it seems!) have also built a geolocation service however mine is built on top of MaxMind's DBs that are mentioned in the post. Its on a few boxes running OpenResty and now handles 130m+ requests/day. Was really fun to build! https://github.com/jloh/geojs https://github.com/jloh/geojs https://www.geojs.io/open/ https://www.geojs.io/open/
- rvnx 6y ago
  It's forbidden by MaxMind ToS, these dudes changed the rules and forces you to abide to the new conditions when you download an update. It's very borderline from them (at least ethically)
  - foota 6y ago
    I didn't see anything in their ToS that would prohibit hosting an API as long as you enforce the same requirements as maxmind does?
  - cpburns2009 6y ago
    At the bottom of the page it credits the MaxMind GeoLite database. The GeoLite2 database is licensed as CC BY-SA 4.0. It's the GeoIP2 database that has stricter licensing. https://dev.maxmind.com/geoip/geoip2/geolite2/ https://dev.maxmind.com/geoip/geoip2/geolite2/
  - itsjloh 6y ago
    As another reply states I publish that the source of data is indeed MaxMind. I agree ethically it doesn't quite sit right so thats why I don't charge for the service.
- deleted 6y ago
  [deleted]
csunbird 6y ago
Interesting, so basically you are making Cloudflare's geoip service public for free.
- EE84M3i 6y ago
  You can do this on appengine too, it provides the client location in a header.
deleted 6y ago
[deleted]
sudhirj 6y ago
Google App Engine had this quite a while ago (almost a decade, I think)and a very generous free tier. Also has approx lat-long, city, state and country. https://blip.runway7.net https://blip.runway7.net https://github.com/runway7/blip https://github.com/runway7/blip I lost track of the consumer apps I've used this on and still haven't received a $1+ bill.
- sneak 6y ago
  Is there a good lat/lon to timezone database that lets you leverage this into a presumptive timezone offset from UTC and thus local time?
  - imustbeevil 6y ago
    Do you need the lat/lon for this? Usually you'd just put in the nearest city. Moment.js is probably the most prevalent time and date library.
  - deleted 6y ago
    [deleted]
  - deanishe 6y ago
    Can’t you just ask the client which timezone it’s in? Every browser knows, and it will be a more accurate answer: the user might be on a foreign VPN or temporarily in another TZ, but not working on that timezone.
    - sneak 6y ago
      My application is not browser-based. I’d like to set a system clock timezone via geoip only, without location lookup via wifi. On my more secure systems I have location services disabled and it’d be nice to have an accurate local time automatically.
      - sroussey 6y ago
        I have a need for this too, and considering making a basic service for it.
waffl 6y ago
I had to tackle this problem as well for a small project and found that I could just send a GET request to https://www.cloudflare.com/cdn-cgi/trace https://www.cloudflare.com/cdn-cgi/trace and parse the response with a simple regex: /loc=(\w*)$/gm The response time averages 10ms and is free, granted I'm not sure if there is a ToS or anything attached to this endpoint, I've only found non-conclusive discussion here https://community.cloudflare.com/t/what-are-the-terms-of-use-for-cdn-cgi-trace/147936 https://community.cloudflare.com/t/what-are-the-terms-of-use...
- jgrahamc 6y ago
  I mean, you can do that but it's a hideous hack and we might change that output at any time and you could easily hit DDoS mitigation or other security features if you do that. That endpoint is used for debugging and is in no way an API. I'd advise you to just create a Cloudflare Worker if you want do this. You'll get more reliability etc. and Workers have a very generous free tier.
- maple3142 6y ago
  It is worth mentioning that /cdn-cgi/trace doesn't only work under www.cloudflare.com. It works for any https?://DOMAIN_WITH_CLOUDFLARE/cdn-cgi/trace . It is an easy to determine whether a website is using CloudFlare or not too.
walrus01 6y ago
since this article mentions maxmind, this reminded me of when geolocation by IP address goes horribly wrong, and non-technical persons interpret the results as something to be relied upon as factual: https://arstechnica.com/tech-policy/2016/08/kansas-couple-sues-ip-mapping-firm-for-turning-their-life-into-a-digital-hell/ https://arstechnica.com/tech-policy/2016/08/kansas-couple-su... https://www.theguardian.com/technology/2016/aug/09/maxmind-mapping-lawsuit-kansas-farm-ip-address https://www.theguardian.com/technology/2016/aug/09/maxmind-m... https://mashable.com/2016/08/11/ip-addresses-kansas/ https://mashable.com/2016/08/11/ip-addresses-kansas/ ISP perspective here: Geolocation by granular /24 to /20 sized block of ipv4 space is often wildly inaccurate on a regional basis. It's entirely possible for the ARIN registration (used by maxmind) to be a street address in Seattle, but serving end user ISP customers a 4.5 hour drive away in a far eastern corner of WA state.
Nextgrid 6y ago
Why would you need to depend on a third-party service and network being available for such a basic task? MaxMind provides their GeoLite database for free and it's extremely easy to embed it in your app.
tuananh 6y ago
it's right here in the worker example https://developers.cloudflare.com/workers/templates/pages/country_code/ https://developers.cloudflare.com/workers/templates/pages/co...
fs2 6y ago
I've used the free MaxMinds database for a while but since the last year I've been using iplist.cc. It supports IPv4, IPv6, shows if an IP is tor, spam, which ASN and a lot more. It's also free and fast but with services like these I always wonder how long they manage to stay free.
- nkozyra 6y ago
  Right. The first worry is "will this be here tomorrow?" The second is "how much more will they end up charging?" I had good success with maxmind's free database. I can't find tooth results for accuracy but my sampling was pretty good. They also have an incentive to keep it.
efesak 6y ago
Also please see https://github.com/analogic/ipgeo https://github.com/analogic/ipgeo daily actualized ip/country database with open license (shameless ad)
- jiofih 6y ago
  That would be a lot more appealing if it explained where the data is sourced from, included the update scripts, and how it is licensed - the only issue asking that question was closed without a word.
  - XCSme 6y ago
    The issue was closed with a commit adding the MIT license to the repo.
  - ColdHeat 6y ago
    It likely uses the data published by RiRs. I wrote something similar that discusses where it sources the data and how to generate it. https://github.com/geoacumen/geoacumen-country https://github.com/geoacumen/geoacumen-country I've been meaning to make automatic Github releases for it.
fasteo 6y ago
Slightly off-topic. AFAIK, an unresolved problem is a proper geolocation service - at least for city level resolution - for mobile IP addresses. There are some services in this field (digital element), but they are very unreliable.
stereo 6y ago
That country list doesn't have Kosovo (2008) or South Sudan (2011).
- dbuxton 6y ago
  Also the republic of north Macedonia is still in there by old name ;)
- jacobush 6y ago
  Kosovo is not universally recognised though, to be fair. But yes, I'd include it. Even if you are against in principle, it's de facto a state and treated as such even by many countries not recognising it in name.
  - hansitomani 6y ago
    But that would mean that your API should return a country specific result to be accurate.
    - AdrianoKF 6y ago
      Which is precisely what for example Mapbox does - they allow consumers to specify a worldview, which adjusts borders and other geographic features based on the specified culture: https://docs.mapbox.com/help/glossary/worldview/ https://docs.mapbox.com/help/glossary/worldview/
      - abafazi 6y ago
        They're missing a Nazi Germany view :(
      - jaakl 6y ago
        They dont have Serbian one to resolve this issue. Or even more relevant ones like Russia (remember Crimea).
- seszett 6y ago
  A somewhat related problem is that overseas regions (at least for France) are handled as different countries. It makes sense for a CDN, but for most other purposes when I use Internet when I'm on Réunion island my country is "FR", not "RE", since RE is not actually a country. It's also quite inconsistent, I've tried with several IPs from Réunion and some of them are FR while the others are RE. It doesn't seem to have the same problem with Hawaii for some reason, they're all just US from what I can see.
gitgud 6y ago
Side note: 6 million requests per month is only 2.2 requests per second, a raspberry PI could do this (technically).
- stormdennis 6y ago
  Would they all be spaced out evenly though?
  - Moru 6y ago
    If the processing time is much below a second per request, requests per second might be more interesting than to stretch it out on a month.
- gitanovic 6y ago
  That's only true if you assume a even distribution. If you cluster them the scenario is a bit different (your conclusion might still hold anyway)
  - doctor_eval 6y ago
    Even if all requests were clustered into one hour per day, it’s only 54 requests per second. Pretty sure a Pi would be up to the job.
    - louwrentius 6y ago
      Yes, HN hit my blog on a 3b+ and that was no issue.
      - SkyLinx 6y ago
        Which blog out of curiosity and 3b requests in which period of time? Is your blog static? Thanks!
        ta17711771 6y ago
skuhn 6y ago
Since most CDNs are already doing GeoIP lookups (for request headers or log entries), you can leverage that to provide the data back in the response body via origin, worker or even CDN edge config. Programmatically populating the response body, as in the Cloudflare worker example from the post, is better than going to the origin just to echo some headers back in the response. To me, something like Fastly's VCL config language is even simpler. It directly executes on every CDN edge node worldwide upon request. For example, I just whipped this up on Fastly using VCL. It returns GeoIP as json data for your IP at the root path: http://geo.zombe.es http://geo.zombe.es Or if you want a particular IP, just append it to the path: http://geo.zombe.es/2a04:4e42:600::313 http://geo.zombe.es/2a04:4e42:600::313 You could do the same via query params, headers, etc. Have URL endpoints that only return some of the data, and so forth. The VCL syntax gets a little gross when you handle quoting strings and assembling json and testing if the string is empty, but it gets the job done. Of course what you might want from GeoIP data may not be what you get. It's really kind of a useful kludge that gets treated sometimes as a panacea. This dataset right now thinks that I'm about 5 miles east of my location, but when subnets are repurposed it could be much more significant. And the data sources are always changing, so who knows what it will think tomorrow.
- jgrahamc 6y ago
  What does the code for this look like?
  - elorm 6y ago
    Here you go. https://gist.github.com/simonkuhn/a380a6fa205db87a3625f26ad09e6593 https://gist.github.com/simonkuhn/a380a6fa205db87a3625f26ad0...
  - skuhn 6y ago
    The VCL for it is available at https://gist.github.com/simonkuhn/a380a6fa205db87a3625f26ad09e6593 https://gist.github.com/simonkuhn/a380a6fa205db87a3625f26ad0... vcl_recv and vcl_error are the important bits, the rest is VCL boilerplate from https://developer.fastly.com/learning/vcl/using/ https://developer.fastly.com/learning/vcl/using/ for unused subroutines.
layoutIfNeeded 6y ago
Can you make a geolocation service from static files? 2^32 IP4 addresses with 2 floats per address would take only 34 gigabytes of storage. Put 256 addresses in a given file, and turn the other three octets into folders. E.g. https://example.com/192/168/0.txt https://example.com/192/168/0.txt would contain the location for addresses 192.168.0.0-192.168.0.255. Would this be cheaper than running these services?
- hansitomani 6y ago
  You could also generate all those files, put that into a GCS Bucket and put an API Gateway in front. easier would be and thats just a thought: You could build a go/c++/java app which contains that database pre compressed in some arbitrary better format than just 2 floats, get it down from your original 34gig and just keep it in memory. Like if you know that 99% of all ips in a certain range are from US, then only store the 1% as a list.
  - layoutIfNeeded 6y ago
    Sure, but my point was that it can be done with zero compute cost. There's nothing cheaper than serving static files from cloud storage.
    - hansitomani 6y ago
      That is just not true. GCS for example and its the same with S3, costs space, egress and requests themself. Data Storage 50 GB Standard Storage * $0.026 per GB $1.30 Network 10 GB egress * $0.12 per GB $1.20 Operations 10,000 Class A operations * $0.05 per 10,000 operations $0.05 Operations 50,000 Class B operations * $0.004 per 10,000 operations $0.02
- zrail 6y ago
  MaxMind sells their databases as files as well as APIs. GeoLite is free to use but not as accurate as their paid products.
mcculley 6y ago
Regarding latency of AWS Lambda: > on average the response took somewhere between from 200ms to 500ms I'm getting latency of 66ms to 126ms with some simple Java code running on AWS Lambda using provisioned concurrency. I find the latency is just fine for most use cases.
- vp8989 6y ago
  "using provisioned concurrency" What's the point of using Lambda if you have to provision capacity for it? You pay a premium on the compute because it's able to scale down to zero (so in theory, it works out cheaper anyway), but in reality it only works well when you do everything in your power to have it not scale down to zero.
  - mcculley 6y ago
    For my use case, it is still much cheaper than having a dedicated VM.
rightbyte 6y ago
The cheapest alternative has to be to ask the users where they are and store it on the user device? I never understood the need for geoip unless you ship spyware. If the exact location is important geoip is not accurate enough anyway. Forwarding to regional sites automatically is just annoying when it doesn't work properly or someone is traveling abroad.
- abafazi 6y ago
  1) Users don't want to be asked where they are 2) GeoIP has plenty of uses that go beyond spyware
- davewritescode 6y ago
  If it's just to route to regional sites, that's fine. Letting the user select just makes sense. However, there's a lot of use cases that semi-accurate geo-locations make sense. The first of which is analytics. If I'm a marketing person at a SaaS company, I want to know where my customers physically are if possible for a variety of useful reasons. Good quality estimates for location also help with security and compliance use cases as well. If a user logs in from a new country on the opposite of the world, you can flag that and take whatever action you want whether it's to block them, fire off an email to the person who owns the account or whatever.
  - sneak 6y ago
    > If I'm a marketing person at a SaaS company, I want to know where my customers physically are if possible for a variety of useful reasons. What reasons? What makes you feel entitled to that information if not volunteered?
    - jaywalk 6y ago
      Any info that can be gleaned from the request is fair game.
  - reaperducer 6y ago
    I want to know where my customers physically are if possible for a variety of useful reasons. I hope you're factoring in error rates. Because right now Google, with its billions of dollars to spend on geolocation, tells me that my laptop is in Albuquerque, and my phone is in Los Angeles. Neither are within 500 miles of either.
sheeshkebab 6y ago
This only work’s if geocoder is accessed directly from browser. It will not if you need to geocode ip addresses received in other ways on the server side. I guess it’s ok for a basic consumer website, although it’s not exactly equivalent to ip geocoder databases/services - they allow passing ip addresses as part of geocoding request.
priyaaank 6y ago
https://ifconfig.co/ https://ifconfig.co/
- dt3ft 6y ago
  I got this: "Application error An error occurred in the application and your page could not be served. If you are the application owner, check your logs for details. You can do this from the Heroku CLI with the command heroku logs --tail"
teknopaul 6y ago
that two per second if you don't have a calculator handy.
sradman 6y ago
TECHNIQUE: use the proprietary HTTP Request headers available through CDN/Cloud providers like Cloudflare Workers' cf-country [1], Amazon CloudFront's CloudFront-Viewer-Country [2], and Google App Engine's X-Appengine-Country/-Region/-City/... [3] to get client Geolocation data. [1] https://developers.cloudflare.com/workers/reference/apis/request/ https://developers.cloudflare.com/workers/reference/apis/req... [2] https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-headers-behavior https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope... [3] https://cloud.google.com/appengine/docs/standard/go/reference/request-response-headers https://cloud.google.com/appengine/docs/standard/go/referenc...
- ErikAugust 6y ago
  Thank you - I was about to ask if this sort of information was available through other cloud providers.
ing33k 6y ago
doing a geo based redirection was my first golang project that got deployed to prod. Nginx + GeoIP2 module sets a HTTP request header and proxies the request to the golang app. The go app does a lookup from redis, based a combination of country header and some url params and responds back with a redirect header. Hosted it on a t2 medium in EC2 and I have seen it easily handle ~1500 requests / second without any issue.
tastytacos 6y ago
I applaud your efforts but it's disingenuous to title this how you "made a geolocation service" when you are just reading a field in someone else's geolocation service.
- georgyo 6y ago
  Indeed. Using another service and being a wrapper around it is not all that impressive. He is also using workers which makes his solution expensive. I run ifconfig.io which now gets just over a billion hits a day. It is basically an echo service, and it just parrots back what cloudflare tells it. But since I run it on linode, it costs me a whopping $40 dollars a month for 35 billion returns a month. Using workers would cost me several thousand dollars. http://ifconfig.io/country_code http://ifconfig.io/country_code
  - joeraut 6y ago
    Great service, thanks for running it!
  - mongo-db-hater 6y ago
    Thanks for sharing. This is a perfect example which proves lambda/serverless/cf worker are snake oils.
    - dclusin 6y ago
      I think there's situations where Lambdas make sense in spite of their pricing. I applied for a job on the Warren campaign and their tech stack relied on AWS lambdas. I think that makes sense for them. It's a short-lived endeavor that isn't going to be around next year. Spending a bit more to not have to deal with the headache of maintaining your own infra makes sense in scenarios like that.
      - 838hhh 6y ago
        You assume that they made a correct and rational decision but you have no evidence to back it up. Retract your claim that there are situations where Lambdas make sense until you can provide concrete examples. Thanks.
contravariant 6y ago
I'm a bit confused. This looks like a few hundred lines of code to read a value from a hardcoded dictionary. Even as a proof of concept it would be more sensible to just add two numbers or something, at least that gives the impression that you could also make the API do something useful.
sigmonsays 6y ago
i read the title and got interested, glad I skimmed. This is very misleading. You didn't "make" a service, you used someone elses.
ignoramous 6y ago
Full marks to Cloudflare for engineering a radically effective and simple alternative to AWS Lambda (+API Gateway): It is simply a fantastic Serverless offering for low-latency network-bound workloads. For paying customers, they even thrown in freemium access to their globally distributed KV store and a forever-free Zonal Cache to sweeten the already good enough deal. That said, good luck with their Support team in case you discover undocumented limits (in production) like these [0]. I got sent a (frustratingly incorrect) bot reply to a ticket and a reminder that Enterprise / Business / Pro customers are priority (in that order) even though I pay for Workers. It has been an uphill battle to get someone to take a look at the ticket so far. Thankfully, we haven't gone to production yet, but as a consequence, now need to plan to add mitigation in scenarios where Workers blacks-out our traffic (but Support can't be of immediate help because "free customer"). [0] https://community.cloudflare.com/t/workers-and-sub-requests/151829 https://community.cloudflare.com/t/workers-and-sub-requests/...
- vasco 6y ago
  > now need to plan to add mitigation in scenarios where Workers blacks-out our traffic (but Support can't be of immediate help because "free customer"). How is it that you prefer to engineer those workarounds instead of just paying $20/month? It surely can't be worth in engineering hours and ongoing maintenance of hacky workaround code?
  - ignoramous 6y ago
    > How is it that you prefer to engineer those workarounds instead of just paying $20/month? If it isn't clear, there's an issue of trust here, not money. And frankly, a fallback shouldn't be much of an issue since fly.io, AWS Lambda@Edge are like-for-like swaps, so we'd still be "serverless" just on a different yet similar platform (albeit costlier).
ColdHeat 6y ago
I've been trying to build open source MaxMind alternatives for a bit. (Mostly so I could distribute them with my open source projects) IP to country is fairly easy and I open sourced all the scripts and the database itself [0]. But IP to city is much harder, I'm not actually sure it's viable for anyone to do that without relying on some other 3rd party service. I'd be very interested to hear if anyone knows how to pull that off in an open sourceable manner. [0] https://github.com/geoacumen/geoacumen-country https://github.com/geoacumen/geoacumen-country
- unilynx 6y ago
  A big part of why you can't freely distribute Maxmind's Lite databases is them trying to apply CCPA requirements. That law will affect open source solutions just as much.
  - ColdHeat 6y ago
    IANAL but Maxmind has to comply with CCPA I assume because they are sourcing data from California residents. If data is sourced in an aggregate or anonymous way, it wouldn't be subject to CCPA. > Aggregate and anonymous data is exempt from the CCPA, unless it is in any way re-identifiable. https://www.cookiebot.com/en/what-is-ccpa/ https://www.cookiebot.com/en/what-is-ccpa/
GiantSully 6y ago
Though a little bit off-topic, before the service http://freegeoip.net/json/ http://freegeoip.net/json/ ceased, I used it a lot in testing. I built one myself with the simplified source code from https://github.com/voyagin/freegeoip https://github.com/voyagin/freegeoip since I want to minimize the response time. BTW, the listed repo isn't mine.
- andruby 6y ago
  I run a hosted version on https://freegeoip.whoisbot.io https://freegeoip.whoisbot.io but it looks like MaxMind changed their database url so it stopped working
freelancercv 6y ago
You can do this for FREE https://freelancer.freelancercv.com/blog/9/find-website-visitors-country-location-using-cloudflare-free-service https://freelancer.freelancercv.com/blog/9/find-website-visi...
hluska 6y ago
All possibly criticisms aside, this is really good and useful. Great job friend!!! It's a beautiful day for hacking!! :)
jamesponddotco 6y ago
I built a simple API using pure NGINX to get the IP address of a client, for times when I needed to ask a customer for his IP address; it was easier sending them a URL for a service I control, than explaining how they could get that information another way. Now I have been thinking about opening it up for more people because while there are a variety of these services out there, one more does not hurt — it already exists, and works, so who knows, maybe more people would like to use it. The code is open, and access logs go to /dev/null; I could probably add a read-only SSH user for people to confirm that for themselves. Geolocation could easily be added to it, but then comes my question: is there any use-case for geolocation APIs that does not involve tracking users for shitty purposes? I was excited about adding this feature because I am using pure NGINX for it, and it was a fun learning experience, but I asked myself that question when I started writing the documentation for the website, and I still do not have an answer. Marketing material for other APIs that offer geolocation usually have user tracking as a selling point. Personally, I have no use for geolocation, and if all use-cases involve tracking without consent users, and breaking their privacy, I want no part in that.
- jwdavidson 6y ago
  I have exactly one non-shady use case: I work with an event org with a fairly common name, and another city in our region has an event with the same name. For some years, we’ve outranked them for major keywords and getting their confused customers was becoming a headache for us. So now if our website thinks you’re in the other city, a little banner appears above the content asking if you’re looking for the other site, and offers the link. This has saved us ~10 phone calls per day in season. Before adding this link we all got to the point of just doing their customer service for them and helping people buy tickets because it was less drama than trying to get them to call the right people.
  - jamesponddotco 6y ago
    While I agree that your use-case is not shady, I am not sure how I would work that in the marketing material for the website — maybe I just suck at writing marketing copy. But in any case, your response does tell me that there are other things that can be done with that information, so maybe I should focus on the IP thing, and have the geolocation features kinda hidden away in the documentation pages.
scottndecker 6y ago
MaxMind data accurate to the zip code level only about 30-60% of the time (as compared with what Google's Geolocation service will provide which is based off more data points than just IP address). Only use MaxMind if you're looking for region or country level accuracy.
ignoramous 6y ago
One of the main challenges of building on Serverless platforms is rate-limiting. There's nothing stopping a script-kiddie from thundering away at the Serverless endpoint resulting in an unexpected and quite high billables [0]. As for Workers specifically, Cloudflare's rate-limiting plan makes the whole thing 10x expensive at $10 for 2,000,000 good-requests [1] + $1 for 2,000,000 Workers requests. Other cloud providers I don't think fare any different. [0] https://community.cloudflare.com/t/how-to-protect-cloudflare-worker-from-ddos/179017 https://community.cloudflare.com/t/how-to-protect-cloudflare... [1] https://support.cloudflare.com/hc/en-us/articles/115000272247 https://support.cloudflare.com/hc/en-us/articles/11500027224...
llaolleh 6y ago
This title is so misleading. I got to the end of the article and thought - wait this is it?!

340 points by maxko 6y ago

crispyporkbites 6y ago

Are you charging for this?

maxko 6y ago
The code is available for free and you can host it on your own Cloudflare account. Cloudflare provides a free plan (up to 100k daily requests across all your Workers), and paid plan ($0.50 per 1m requests)

PaywallBuster 6y ago

tl;dr uses Cloudflares workers as API endpoint and returns Country name based on header in the CF request.

bellwether 6y ago

Love the ingenuity and thank you for the performance comparison on Cloudflare Workers vs AWS Lambda. I personally wouldn’t consider what you built a Geolocation service, but glad it solved your use case!

maxko 6y ago
Thanks!

speedgoose 6y ago

Well that's a bit more than 2 requests per second on average. Sure you may have some bursts once in a while, but nothing a 3€ VM can't manage IMHO.

Nextgrid 6y ago
I always find it funny when people brag about cost savings on cloud services while overlooking the insane premium they're paying by using the "cloud" to begin with. It's kinda like a construction firm using a supercar to haul materials and then bragging about their efforts to make cosmetic repairs cost-effective... maybe you could just not use an overpriced car to begin with and then damage won't be an issue?
- harpratap 6y ago
  Am I missing something? Isn't a $3 VM referring to a VM service in a cloud somewhere? If you try to do the same thing without cloud you'll have to end up renting space in a datacenter, install your own servers, connect to an ISP, power bills, etc etc. Cloud definitely seems cheaper in comparison
  - progval 6y ago
    GP and you aren't using the same definition of cloud. By cloud, they mean infrastuctures like AWS/GCP/Azure, not simple VM hosting (called VPS outside "clouds", and compute nodes in "clouds")
    - harpratap 6y ago
      What's the definition of "cloud" then which excludes VPS from the umbrella of services offered in cloud? Not really sure how a VPS is any different from a Google Compute Engine
      - XCSme 6y ago
        I think the main difference is the pricing structure. Usually "cloud" refers to the auto-scaling servers with the pay-as-you go pricing, while VPSs are usually manually instantiated and have a fixed monthly cost. Also, sometimes with VPSs you get more control over the server's settings than with "cloud" services. So "cloud" is like renting a dynamically changing number of VPSs each month and which are usually more costly if your consumption is not spiky (eg. when a cheap VPS rented 24/7 is enough).
- dmortin 6y ago
  The main advantage of the cloud for me that it provides PaaS services. You don't have to administer a VM, etc, you just use the services and it's someone else's job to install security updates and stuff.
  - 6y ago
csunbird 6y ago
A Pi should be able to handle this too.
- quickthrower2 6y ago
  A Pi is basically yesteryear’s beefy rack server.
jbverschoor 6y ago
The only thing I like about lambda, is that your code could easily be deployed to different geo locations
- akx 6y ago
  You can also easily spin up a VM in a different geographic location.
  - luckylion 6y ago
    That's true, but likely not as close to the customer as CF Workers on edge. And certainly not as easy, as CF does all that automatically.
  - eat_veggies 6y ago
    If you spun up a VM in 200 locations like Cloudflare, that $3 VPS suddenly becomes $600
mikeyjk 6y ago
A lambda is potentially cheaper if there are long periods with no execution though?

pereiratr 6y ago

Nice! Have you tried AWS Lambda@Edge?

pereiratr 6y ago
and it looks like similar geolocation info "cloudfront-viewer-country" is available from AWS as well. So similar trick maybe possible with AWS https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-examples.html#lambda-examples-redirect-based-on-country https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

wingi 6y ago

I had a geo-location service running in google app engine. for free. After pimping caching the daily traffic of 400.000 requests/day was in the free-tier. https://www.united-coders.com/christian-harms/detailed-performance-optimization-steps-on-webservice/ https://www.united-coders.com/christian-harms/detailed-perfo...

jasonlingx 6y ago

This one I made couple years ago (and haven’t checked in years) is still running for free thanks to Heroku and Cloudflare: https://github.com/jlxw/geoip https://github.com/jlxw/geoip

whatl3y 6y ago

So this finds the country based on some cloudflare-populated info of an incoming request (which sounds like it solved OP’s problem which is good), but if you want to use MaxMind’s database to find the country of any public IP I built the following a few months back: https://github.com/Risk3sixty-Labs/geoapi https://github.com/Risk3sixty-Labs/geoapi

EE84M3i 6y ago

What are the runtime limitations that prevent the maxmind database from working inside cloudflare workers?

achairapart 6y ago
I was wondering the same.

grizzles 6y ago

How does MaxMind prevent someone from releasing an open source version of their database? If you are about to answer "Copyright", remember - you can't Copyright facts. This has been upheld in the Courts system many times.

itsjloh 6y ago
MaxMind have two versions of their database, the free to use version called GeoLite and the paid version (whos name I'm not aware of). I realise this doesn't directly answer your question but I guess one reason might be that they already provide a free version so releasing the paid version would just get you in strife?
deleted 6y ago
[deleted]
mike_d 6y ago
Facts cannot be copyrighted, but a unique compilation of facts can be. This became settled law many decades ago with sports almanacs. Furthermore, Maxmind adds non-factual entries to the database so they can identify it uniquely as their work. Map makers do something similar: https://en.wikipedia.org/wiki/Trap_street https://en.wikipedia.org/wiki/Trap_street
renewiltord 6y ago
You can just stick this in a license agreement, surely.
Mindwipe 6y ago
Maxmind's database is updated very regularly and includes a lot of things like risk scoring. Those are absolutely protected.
redis_mlc 6y ago
> If you are about to answer "Copyright", remember - you can't Copyright facts. Nope, US law was changed over a decade ago.

itsjloh 6y ago

I (like many others it seems!) have also built a geolocation service however mine is built on top of MaxMind's DBs that are mentioned in the post. Its on a few boxes running OpenResty and now handles 130m+ requests/day. Was really fun to build! https://github.com/jloh/geojs https://github.com/jloh/geojs https://www.geojs.io/open/ https://www.geojs.io/open/

rvnx 6y ago
It's forbidden by MaxMind ToS, these dudes changed the rules and forces you to abide to the new conditions when you download an update. It's very borderline from them (at least ethically)
- foota 6y ago
  I didn't see anything in their ToS that would prohibit hosting an API as long as you enforce the same requirements as maxmind does?
- cpburns2009 6y ago
  At the bottom of the page it credits the MaxMind GeoLite database. The GeoLite2 database is licensed as CC BY-SA 4.0. It's the GeoIP2 database that has stricter licensing. https://dev.maxmind.com/geoip/geoip2/geolite2/ https://dev.maxmind.com/geoip/geoip2/geolite2/
- itsjloh 6y ago
  As another reply states I publish that the source of data is indeed MaxMind. I agree ethically it doesn't quite sit right so thats why I don't charge for the service.
deleted 6y ago
[deleted]

csunbird 6y ago

Interesting, so basically you are making Cloudflare's geoip service public for free.

EE84M3i 6y ago
You can do this on appengine too, it provides the client location in a header.

deleted 6y ago

[deleted]

sudhirj 6y ago

Google App Engine had this quite a while ago (almost a decade, I think)and a very generous free tier. Also has approx lat-long, city, state and country. https://blip.runway7.net https://blip.runway7.net https://github.com/runway7/blip https://github.com/runway7/blip I lost track of the consumer apps I've used this on and still haven't received a $1+ bill.

sneak 6y ago
Is there a good lat/lon to timezone database that lets you leverage this into a presumptive timezone offset from UTC and thus local time?
- imustbeevil 6y ago
  Do you need the lat/lon for this? Usually you'd just put in the nearest city. Moment.js is probably the most prevalent time and date library.
- deleted 6y ago
  [deleted]
- deanishe 6y ago
  Can’t you just ask the client which timezone it’s in? Every browser knows, and it will be a more accurate answer: the user might be on a foreign VPN or temporarily in another TZ, but not working on that timezone.
  - sneak 6y ago
    My application is not browser-based. I’d like to set a system clock timezone via geoip only, without location lookup via wifi. On my more secure systems I have location services disabled and it’d be nice to have an accurate local time automatically.
    - sroussey 6y ago
      I have a need for this too, and considering making a basic service for it.

waffl 6y ago

I had to tackle this problem as well for a small project and found that I could just send a GET request to https://www.cloudflare.com/cdn-cgi/trace https://www.cloudflare.com/cdn-cgi/trace and parse the response with a simple regex: /loc=(\w*)$/gm The response time averages 10ms and is free, granted I'm not sure if there is a ToS or anything attached to this endpoint, I've only found non-conclusive discussion here https://community.cloudflare.com/t/what-are-the-terms-of-use-for-cdn-cgi-trace/147936 https://community.cloudflare.com/t/what-are-the-terms-of-use...

jgrahamc 6y ago
I mean, you can do that but it's a hideous hack and we might change that output at any time and you could easily hit DDoS mitigation or other security features if you do that. That endpoint is used for debugging and is in no way an API. I'd advise you to just create a Cloudflare Worker if you want do this. You'll get more reliability etc. and Workers have a very generous free tier.
maple3142 6y ago
It is worth mentioning that /cdn-cgi/trace doesn't only work under www.cloudflare.com. It works for any https?://DOMAIN_WITH_CLOUDFLARE/cdn-cgi/trace . It is an easy to determine whether a website is using CloudFlare or not too.

walrus01 6y ago

since this article mentions maxmind, this reminded me of when geolocation by IP address goes horribly wrong, and non-technical persons interpret the results as something to be relied upon as factual: https://arstechnica.com/tech-policy/2016/08/kansas-couple-sues-ip-mapping-firm-for-turning-their-life-into-a-digital-hell/ https://arstechnica.com/tech-policy/2016/08/kansas-couple-su... https://www.theguardian.com/technology/2016/aug/09/maxmind-mapping-lawsuit-kansas-farm-ip-address https://www.theguardian.com/technology/2016/aug/09/maxmind-m... https://mashable.com/2016/08/11/ip-addresses-kansas/ https://mashable.com/2016/08/11/ip-addresses-kansas/ ISP perspective here: Geolocation by granular /24 to /20 sized block of ipv4 space is often wildly inaccurate on a regional basis. It's entirely possible for the ARIN registration (used by maxmind) to be a street address in Seattle, but serving end user ISP customers a 4.5 hour drive away in a far eastern corner of WA state.

Nextgrid 6y ago

Why would you need to depend on a third-party service and network being available for such a basic task? MaxMind provides their GeoLite database for free and it's extremely easy to embed it in your app.

tuananh 6y ago

it's right here in the worker example https://developers.cloudflare.com/workers/templates/pages/country_code/ https://developers.cloudflare.com/workers/templates/pages/co...

fs2 6y ago

I've used the free MaxMinds database for a while but since the last year I've been using iplist.cc. It supports IPv4, IPv6, shows if an IP is tor, spam, which ASN and a lot more. It's also free and fast but with services like these I always wonder how long they manage to stay free.

nkozyra 6y ago
Right. The first worry is "will this be here tomorrow?" The second is "how much more will they end up charging?" I had good success with maxmind's free database. I can't find tooth results for accuracy but my sampling was pretty good. They also have an incentive to keep it.

efesak 6y ago

Also please see https://github.com/analogic/ipgeo https://github.com/analogic/ipgeo daily actualized ip/country database with open license (shameless ad)

jiofih 6y ago
That would be a lot more appealing if it explained where the data is sourced from, included the update scripts, and how it is licensed - the only issue asking that question was closed without a word.
- XCSme 6y ago
  The issue was closed with a commit adding the MIT license to the repo.
- ColdHeat 6y ago
  It likely uses the data published by RiRs. I wrote something similar that discusses where it sources the data and how to generate it. https://github.com/geoacumen/geoacumen-country https://github.com/geoacumen/geoacumen-country I've been meaning to make automatic Github releases for it.

fasteo 6y ago

Slightly off-topic. AFAIK, an unresolved problem is a proper geolocation service - at least for city level resolution - for mobile IP addresses. There are some services in this field (digital element), but they are very unreliable.

stereo 6y ago

That country list doesn't have Kosovo (2008) or South Sudan (2011).

dbuxton 6y ago
Also the republic of north Macedonia is still in there by old name ;)
jacobush 6y ago
Kosovo is not universally recognised though, to be fair. But yes, I'd include it. Even if you are against in principle, it's de facto a state and treated as such even by many countries not recognising it in name.
- hansitomani 6y ago
  But that would mean that your API should return a country specific result to be accurate.
  - AdrianoKF 6y ago
    Which is precisely what for example Mapbox does - they allow consumers to specify a worldview, which adjusts borders and other geographic features based on the specified culture: https://docs.mapbox.com/help/glossary/worldview/ https://docs.mapbox.com/help/glossary/worldview/
    - abafazi 6y ago
      They're missing a Nazi Germany view :(
    - jaakl 6y ago
      They dont have Serbian one to resolve this issue. Or even more relevant ones like Russia (remember Crimea).
seszett 6y ago
A somewhat related problem is that overseas regions (at least for France) are handled as different countries. It makes sense for a CDN, but for most other purposes when I use Internet when I'm on Réunion island my country is "FR", not "RE", since RE is not actually a country. It's also quite inconsistent, I've tried with several IPs from Réunion and some of them are FR while the others are RE. It doesn't seem to have the same problem with Hawaii for some reason, they're all just US from what I can see.

gitgud 6y ago

Side note: 6 million requests per month is only 2.2 requests per second, a raspberry PI could do this (technically).

stormdennis 6y ago
Would they all be spaced out evenly though?
- Moru 6y ago
  If the processing time is much below a second per request, requests per second might be more interesting than to stretch it out on a month.
gitanovic 6y ago
That's only true if you assume a even distribution. If you cluster them the scenario is a bit different (your conclusion might still hold anyway)
- doctor_eval 6y ago
  Even if all requests were clustered into one hour per day, it’s only 54 requests per second. Pretty sure a Pi would be up to the job.
  - louwrentius 6y ago
    Yes, HN hit my blog on a 3b+ and that was no issue.
    - SkyLinx 6y ago
      Which blog out of curiosity and 3b requests in which period of time? Is your blog static? Thanks!
      - ta17711771 6y ago

skuhn 6y ago

Since most CDNs are already doing GeoIP lookups (for request headers or log entries), you can leverage that to provide the data back in the response body via origin, worker or even CDN edge config. Programmatically populating the response body, as in the Cloudflare worker example from the post, is better than going to the origin just to echo some headers back in the response. To me, something like Fastly's VCL config language is even simpler. It directly executes on every CDN edge node worldwide upon request. For example, I just whipped this up on Fastly using VCL. It returns GeoIP as json data for your IP at the root path: http://geo.zombe.es http://geo.zombe.es Or if you want a particular IP, just append it to the path: http://geo.zombe.es/2a04:4e42:600::313 http://geo.zombe.es/2a04:4e42:600::313 You could do the same via query params, headers, etc. Have URL endpoints that only return some of the data, and so forth. The VCL syntax gets a little gross when you handle quoting strings and assembling json and testing if the string is empty, but it gets the job done. Of course what you might want from GeoIP data may not be what you get. It's really kind of a useful kludge that gets treated sometimes as a panacea. This dataset right now thinks that I'm about 5 miles east of my location, but when subnets are repurposed it could be much more significant. And the data sources are always changing, so who knows what it will think tomorrow.

jgrahamc 6y ago
What does the code for this look like?
- elorm 6y ago
  Here you go. https://gist.github.com/simonkuhn/a380a6fa205db87a3625f26ad09e6593 https://gist.github.com/simonkuhn/a380a6fa205db87a3625f26ad0...
- skuhn 6y ago
  The VCL for it is available at https://gist.github.com/simonkuhn/a380a6fa205db87a3625f26ad09e6593 https://gist.github.com/simonkuhn/a380a6fa205db87a3625f26ad0... vcl_recv and vcl_error are the important bits, the rest is VCL boilerplate from https://developer.fastly.com/learning/vcl/using/ https://developer.fastly.com/learning/vcl/using/ for unused subroutines.

layoutIfNeeded 6y ago

Can you make a geolocation service from static files? 2^32 IP4 addresses with 2 floats per address would take only 34 gigabytes of storage. Put 256 addresses in a given file, and turn the other three octets into folders. E.g. https://example.com/192/168/0.txt https://example.com/192/168/0.txt would contain the location for addresses 192.168.0.0-192.168.0.255. Would this be cheaper than running these services?

hansitomani 6y ago
You could also generate all those files, put that into a GCS Bucket and put an API Gateway in front. easier would be and thats just a thought: You could build a go/c++/java app which contains that database pre compressed in some arbitrary better format than just 2 floats, get it down from your original 34gig and just keep it in memory. Like if you know that 99% of all ips in a certain range are from US, then only store the 1% as a list.
- layoutIfNeeded 6y ago
  Sure, but my point was that it can be done with zero compute cost. There's nothing cheaper than serving static files from cloud storage.
  - hansitomani 6y ago
    That is just not true. GCS for example and its the same with S3, costs space, egress and requests themself. Data Storage 50 GB Standard Storage * $0.026 per GB $1.30 Network 10 GB egress * $0.12 per GB $1.20 Operations 10,000 Class A operations * $0.05 per 10,000 operations $0.05 Operations 50,000 Class B operations * $0.004 per 10,000 operations $0.02
zrail 6y ago
MaxMind sells their databases as files as well as APIs. GeoLite is free to use but not as accurate as their paid products.

mcculley 6y ago

Regarding latency of AWS Lambda: > on average the response took somewhere between from 200ms to 500ms I'm getting latency of 66ms to 126ms with some simple Java code running on AWS Lambda using provisioned concurrency. I find the latency is just fine for most use cases.

vp8989 6y ago
"using provisioned concurrency" What's the point of using Lambda if you have to provision capacity for it? You pay a premium on the compute because it's able to scale down to zero (so in theory, it works out cheaper anyway), but in reality it only works well when you do everything in your power to have it not scale down to zero.
- mcculley 6y ago
  For my use case, it is still much cheaper than having a dedicated VM.

rightbyte 6y ago

The cheapest alternative has to be to ask the users where they are and store it on the user device? I never understood the need for geoip unless you ship spyware. If the exact location is important geoip is not accurate enough anyway. Forwarding to regional sites automatically is just annoying when it doesn't work properly or someone is traveling abroad.

abafazi 6y ago
1) Users don't want to be asked where they are 2) GeoIP has plenty of uses that go beyond spyware
davewritescode 6y ago
If it's just to route to regional sites, that's fine. Letting the user select just makes sense. However, there's a lot of use cases that semi-accurate geo-locations make sense. The first of which is analytics. If I'm a marketing person at a SaaS company, I want to know where my customers physically are if possible for a variety of useful reasons. Good quality estimates for location also help with security and compliance use cases as well. If a user logs in from a new country on the opposite of the world, you can flag that and take whatever action you want whether it's to block them, fire off an email to the person who owns the account or whatever.
- sneak 6y ago
  > If I'm a marketing person at a SaaS company, I want to know where my customers physically are if possible for a variety of useful reasons. What reasons? What makes you feel entitled to that information if not volunteered?
  - jaywalk 6y ago
    Any info that can be gleaned from the request is fair game.
- reaperducer 6y ago
  I want to know where my customers physically are if possible for a variety of useful reasons. I hope you're factoring in error rates. Because right now Google, with its billions of dollars to spend on geolocation, tells me that my laptop is in Albuquerque, and my phone is in Los Angeles. Neither are within 500 miles of either.

sheeshkebab 6y ago

This only work’s if geocoder is accessed directly from browser. It will not if you need to geocode ip addresses received in other ways on the server side. I guess it’s ok for a basic consumer website, although it’s not exactly equivalent to ip geocoder databases/services - they allow passing ip addresses as part of geocoding request.

priyaaank 6y ago

https://ifconfig.co/ https://ifconfig.co/

dt3ft 6y ago
I got this: "Application error An error occurred in the application and your page could not be served. If you are the application owner, check your logs for details. You can do this from the Heroku CLI with the command heroku logs --tail"

teknopaul 6y ago

that two per second if you don't have a calculator handy.

sradman 6y ago

TECHNIQUE: use the proprietary HTTP Request headers available through CDN/Cloud providers like Cloudflare Workers' cf-country [1], Amazon CloudFront's CloudFront-Viewer-Country [2], and Google App Engine's X-Appengine-Country/-Region/-City/... [3] to get client Geolocation data. [1] https://developers.cloudflare.com/workers/reference/apis/request/ https://developers.cloudflare.com/workers/reference/apis/req... [2] https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-headers-behavior https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope... [3] https://cloud.google.com/appengine/docs/standard/go/reference/request-response-headers https://cloud.google.com/appengine/docs/standard/go/referenc...

ErikAugust 6y ago
Thank you - I was about to ask if this sort of information was available through other cloud providers.

ing33k 6y ago

doing a geo based redirection was my first golang project that got deployed to prod. Nginx + GeoIP2 module sets a HTTP request header and proxies the request to the golang app. The go app does a lookup from redis, based a combination of country header and some url params and responds back with a redirect header. Hosted it on a t2 medium in EC2 and I have seen it easily handle ~1500 requests / second without any issue.

tastytacos 6y ago

I applaud your efforts but it's disingenuous to title this how you "made a geolocation service" when you are just reading a field in someone else's geolocation service.

georgyo 6y ago
Indeed. Using another service and being a wrapper around it is not all that impressive. He is also using workers which makes his solution expensive. I run ifconfig.io which now gets just over a billion hits a day. It is basically an echo service, and it just parrots back what cloudflare tells it. But since I run it on linode, it costs me a whopping $40 dollars a month for 35 billion returns a month. Using workers would cost me several thousand dollars. http://ifconfig.io/country_code http://ifconfig.io/country_code
- joeraut 6y ago
  Great service, thanks for running it!
- mongo-db-hater 6y ago
  Thanks for sharing. This is a perfect example which proves lambda/serverless/cf worker are snake oils.
  - dclusin 6y ago
    I think there's situations where Lambdas make sense in spite of their pricing. I applied for a job on the Warren campaign and their tech stack relied on AWS lambdas. I think that makes sense for them. It's a short-lived endeavor that isn't going to be around next year. Spending a bit more to not have to deal with the headache of maintaining your own infra makes sense in scenarios like that.
    - 838hhh 6y ago
      You assume that they made a correct and rational decision but you have no evidence to back it up. Retract your claim that there are situations where Lambdas make sense until you can provide concrete examples. Thanks.

contravariant 6y ago

I'm a bit confused. This looks like a few hundred lines of code to read a value from a hardcoded dictionary. Even as a proof of concept it would be more sensible to just add two numbers or something, at least that gives the impression that you could also make the API do something useful.

sigmonsays 6y ago

i read the title and got interested, glad I skimmed. This is very misleading. You didn't "make" a service, you used someone elses.

ignoramous 6y ago

Full marks to Cloudflare for engineering a radically effective and simple alternative to AWS Lambda (+API Gateway): It is simply a fantastic Serverless offering for low-latency network-bound workloads. For paying customers, they even thrown in freemium access to their globally distributed KV store and a forever-free Zonal Cache to sweeten the already good enough deal. That said, good luck with their Support team in case you discover undocumented limits (in production) like these [0]. I got sent a (frustratingly incorrect) bot reply to a ticket and a reminder that Enterprise / Business / Pro customers are priority (in that order) even though I pay for Workers. It has been an uphill battle to get someone to take a look at the ticket so far. Thankfully, we haven't gone to production yet, but as a consequence, now need to plan to add mitigation in scenarios where Workers blacks-out our traffic (but Support can't be of immediate help because "free customer"). [0] https://community.cloudflare.com/t/workers-and-sub-requests/151829 https://community.cloudflare.com/t/workers-and-sub-requests/...

vasco 6y ago
> now need to plan to add mitigation in scenarios where Workers blacks-out our traffic (but Support can't be of immediate help because "free customer"). How is it that you prefer to engineer those workarounds instead of just paying $20/month? It surely can't be worth in engineering hours and ongoing maintenance of hacky workaround code?
- ignoramous 6y ago
  > How is it that you prefer to engineer those workarounds instead of just paying $20/month? If it isn't clear, there's an issue of trust here, not money. And frankly, a fallback shouldn't be much of an issue since fly.io, AWS Lambda@Edge are like-for-like swaps, so we'd still be "serverless" just on a different yet similar platform (albeit costlier).

ColdHeat 6y ago

I've been trying to build open source MaxMind alternatives for a bit. (Mostly so I could distribute them with my open source projects) IP to country is fairly easy and I open sourced all the scripts and the database itself [0]. But IP to city is much harder, I'm not actually sure it's viable for anyone to do that without relying on some other 3rd party service. I'd be very interested to hear if anyone knows how to pull that off in an open sourceable manner. [0] https://github.com/geoacumen/geoacumen-country https://github.com/geoacumen/geoacumen-country

unilynx 6y ago
A big part of why you can't freely distribute Maxmind's Lite databases is them trying to apply CCPA requirements. That law will affect open source solutions just as much.
- ColdHeat 6y ago
  IANAL but Maxmind has to comply with CCPA I assume because they are sourcing data from California residents. If data is sourced in an aggregate or anonymous way, it wouldn't be subject to CCPA. > Aggregate and anonymous data is exempt from the CCPA, unless it is in any way re-identifiable. https://www.cookiebot.com/en/what-is-ccpa/ https://www.cookiebot.com/en/what-is-ccpa/

GiantSully 6y ago

Though a little bit off-topic, before the service http://freegeoip.net/json/ http://freegeoip.net/json/ ceased, I used it a lot in testing. I built one myself with the simplified source code from https://github.com/voyagin/freegeoip https://github.com/voyagin/freegeoip since I want to minimize the response time. BTW, the listed repo isn't mine.

andruby 6y ago
I run a hosted version on https://freegeoip.whoisbot.io https://freegeoip.whoisbot.io but it looks like MaxMind changed their database url so it stopped working

freelancercv 6y ago

You can do this for FREE https://freelancer.freelancercv.com/blog/9/find-website-visitors-country-location-using-cloudflare-free-service https://freelancer.freelancercv.com/blog/9/find-website-visi...

hluska 6y ago

All possibly criticisms aside, this is really good and useful. Great job friend!!! It's a beautiful day for hacking!! :)

jamesponddotco 6y ago

I built a simple API using pure NGINX to get the IP address of a client, for times when I needed to ask a customer for his IP address; it was easier sending them a URL for a service I control, than explaining how they could get that information another way. Now I have been thinking about opening it up for more people because while there are a variety of these services out there, one more does not hurt — it already exists, and works, so who knows, maybe more people would like to use it. The code is open, and access logs go to /dev/null; I could probably add a read-only SSH user for people to confirm that for themselves. Geolocation could easily be added to it, but then comes my question: is there any use-case for geolocation APIs that does not involve tracking users for shitty purposes? I was excited about adding this feature because I am using pure NGINX for it, and it was a fun learning experience, but I asked myself that question when I started writing the documentation for the website, and I still do not have an answer. Marketing material for other APIs that offer geolocation usually have user tracking as a selling point. Personally, I have no use for geolocation, and if all use-cases involve tracking without consent users, and breaking their privacy, I want no part in that.

jwdavidson 6y ago
I have exactly one non-shady use case: I work with an event org with a fairly common name, and another city in our region has an event with the same name. For some years, we’ve outranked them for major keywords and getting their confused customers was becoming a headache for us. So now if our website thinks you’re in the other city, a little banner appears above the content asking if you’re looking for the other site, and offers the link. This has saved us ~10 phone calls per day in season. Before adding this link we all got to the point of just doing their customer service for them and helping people buy tickets because it was less drama than trying to get them to call the right people.
- jamesponddotco 6y ago
  While I agree that your use-case is not shady, I am not sure how I would work that in the marketing material for the website — maybe I just suck at writing marketing copy. But in any case, your response does tell me that there are other things that can be done with that information, so maybe I should focus on the IP thing, and have the geolocation features kinda hidden away in the documentation pages.

scottndecker 6y ago

MaxMind data accurate to the zip code level only about 30-60% of the time (as compared with what Google's Geolocation service will provide which is based off more data points than just IP address). Only use MaxMind if you're looking for region or country level accuracy.

ignoramous 6y ago

One of the main challenges of building on Serverless platforms is rate-limiting. There's nothing stopping a script-kiddie from thundering away at the Serverless endpoint resulting in an unexpected and quite high billables [0]. As for Workers specifically, Cloudflare's rate-limiting plan makes the whole thing 10x expensive at $10 for 2,000,000 good-requests [1] + $1 for 2,000,000 Workers requests. Other cloud providers I don't think fare any different. [0] https://community.cloudflare.com/t/how-to-protect-cloudflare-worker-from-ddos/179017 https://community.cloudflare.com/t/how-to-protect-cloudflare... [1] https://support.cloudflare.com/hc/en-us/articles/115000272247 https://support.cloudflare.com/hc/en-us/articles/11500027224...

llaolleh 6y ago

This title is so misleading. I got to the end of the article and thought - wait this is it?!