8 ms·
Choosing Go to implement the new GOV.UK router
- justincormack 13y agoAs they are Using Nginx already interested to know if they ruled out openresty or did not consider it.
- buro9 13y agoI'm a big fan of Go and we use it internally, but I too wonder why nginx + Lua wouldn't be the chosen path. If it's good enough for Cloudflare it probably is good enough here. Besides, Varnish also would have excelled (yes I read the first post in the series) if they had just stored the configuration in a saner format and then output the required VCL when the config changed and reloaded on the fly. Both of the technologies I would've picked (nginx and Varnish) were in their stack. I love working in Go, but reading this feels like a reinvention of the wheel. Go would be my choice only if these other things didn't exist already and you had to start from scratch. The biggest pain identified was the writing, maintaining and loading of the router config. Openresty could've solved that.
- intranation 13y agoOpenresty and Lua are great, but Go is a more generally applicable language for building web services. So when taken in that context Go makes more sense if we look at it as a learning exercise as well as a new part of the stack.
- JazCE 13y agoScala wasn't a learning exercise then?
- csmuk 13y agoNot impressed. I have nothing against Go (in fact I like it) but they seem to have built GOV.UK with every damn bit of technology there is available. I've seen Varnish, Go, Ruby, Python, Scala, Java, Mongo, MySQL, nginx, rails, sinatra, Django and PILES of Not Invented Here. Also I'm not sure it's a great use of public money to build stuff like this when they should have nginx/apache up front and some configuration. In fact this whole thing stinks of crappy information architecture resulting in a massive front end router hack that they did in Scala, binned and moved to Varnish, then binned and moved to Go. This is not one "cohesive" web front end. It's a rat's nest and an increasingly expensive one.
- walshemj 13y agoIn their defense having worked on sorting out problems on old complex large scale publishers sites. If your trying to make a collection of services hang together in a seamless way you do need some sort of front end to tie it all together and that needs to be usable by non gurus. Though they do seem to be using all the new toys maybe a bit more thought about which technology to use rather than jumping from technology to technology.
- csmuk 13y agoYep I understand that. We publish several huge legacy applications, several huge new applications, several integrations, public web site, documentation, online support. A mere 80-100 million HTTP transactions a day. Not once have we built something to do it all.
- JazCE 13y agoI tend to agree. The post makes a point of saying "we had a router built in scala, but no one on the team knew how to maintain it". It seems they're allowing team members to play with "language of the week" (lotw), build something that's then used in production without the rest of the team knowing the lotw, that member moving on and then rebuilding it in a new lotw 6 months later. no doubt this GO implementation will go the same way at a later date. I'm not opposed to using a lotw in production with other languages and such, but it needs to be done in a thought out manner with the whole "what if the project lead gets hit by a big red double decker" mentality.
- 13y ago
- shawabawa3 13y agoOn the github page it says the router sits between nginx and varnish and does: * Reverse proxy, forwarding requests to and serving responses from multiple backend servers on a single domain. * Redirector, serving HTTP 301 and 302 redirects to new URLs. * Gone responder, serving HTTP 410 responses for resources that used to but no longer exist. Does anyone know why they have this as a separate piece of software instead of getting nginx to do it?
- rossj 13y agoYou should read the first article in the series - https://gdstechnology.blog.gov.uk/2013/12/05/building-a-new-router-for-gov-uk/ https://gdstechnology.blog.gov.uk/2013/12/05/building-a-new-...
- csmuk 13y agoI actually read that and thought nginx/mod_proxy. Looks like they didn't actually try it or know about it which is worrying. Either that or it was more interesting to do it in Go which is not a valid reason in a taxpayer funded site.
- floops 13y agoDynamic route updating with nginx is not fun.
- csmuk 13y agoNo but updating static routes via puppet/ansible and reloading is a piece of piss. If your site is volatile enough that the routing needs to change that often then there is something wrong either with your information architecture or your development process.
- floops 13y agoOr maybe you're trying to deal with a huge legacy content migration of several hundred different government agencies and want to deploy several times a day without waiting for 10 minutes for nginx to load 1MB of config each time?
- pjmlp 13y agoSo they are ditching Scala, because no one in the team is an expert, while replacing by a language that everyone has to learn?!?
- rossj 13y agoI asked on twitter, and got a reasonable response. Investing in learning Scala is a much longer term thing than learning Go (which is, in their experience, a much quicker thing to do).
- CmonDev 13y agoBecause you always need to use all the features the language provides. Simpler -> limited.
- inglesp 13y agoI'd far, far rather relearn Go than relearn Scala...
- dscrd 13y agoGiven that Scala is far more complex, learning Go from scratch is probably a lesser task.
- hackerboos 13y agoHow did they come to choose Scalatra if nobody knows how to use Scala?
- thu 13y agoThat seems weird to me: * No one having a deep knowledge of Scala is mentioned as a problem, but they will learn Go. * They rule out a language because of its syntax.
- girvo 13y agoI like Erlang, but in shops I've worked with that have your "average" dev in it, it's syntax and semantics would be something they struggle with (if it ain't PHP, they'll struggle with it...). Unfortunately I don't blame them. That said, I think if they truly needed to build a somewhat stateful concurrent router, Erlang is a better choice (solely because it's battle tested in those highly concurrent workloads). Picking tech for big things like this when you have to balance a teams knowledge with the right tool is a hard problem in my experience.
- Zenst 13y agoI agree, but if you want to learn something today then GO would fit better for the task at hand and easier to learn. That and the cost involved is really good value given the training involved with learning Scala over GO. That and I suspect there was some wishlists on what people wanted to do in the mix, like COBOL, still used but people just do not want to learn that as it is deemed so last century in comparsision to say java, though they are altogether different fish. Bit like the difference between a proxy and a router, what sounds better on a CV. I suspect that from a maintenance aspect that the GO code will be much easier to support and grow as and when needed as well as scaleability. Still it is most encouraging to see such a young language like GO get so much respect and use so quickly in a enviroment that has to many in the past and present been slow to adapts new technology. But there again many Govermental departments still have XP, albeit used as a fancy terminal to some old mainframe application that just ticks along.
- floops 13y agoA lot of the comments here seem to be missing something quite important: Scala and Go are not similar languages. Scala is a large and complex language, encompassing multiple programming idioms. Teams programming Scala (as with C++ and many other "large" languages) often have difficulties nailing down what portion of the language they are going to use. "Java-without-semicolons" all the way through to idiomatic, functional Scala. Go is a relatively small, simple language. You could easily learn Go's basic syntax and semantics in an afternoon or two. Simplicity and suitability for programmers is an explicit design goal. Switching away from Scala due to lack of experience while simultaneously picking up Go is not as illogical as everyone seems to be implying.
- JazCE 13y agoYou're right, but i think the point you're missing (and i don't mean this in a rude way) is that they had a system built in scala, somehow, but now no one on the team knows scala. They lack an architectural oversight, that they should really be using one(ish) language rather than a big mix of languages so that this doesn't occur again when no one on the team knows GO because the project leads taken a new job.
- girvo 13y agoRetention. Also, I'd suggest that your average dev will learn Go quick enough to work on this router quicker than they'll learn Scala; I prefer Scala myself, though! It's a balance between proper tech choice, correct management choices moving forward, and dealing with management mistakes (like letting someone build the router in Scala when the rest of the team don't know it) from the past. A crappy PHP dev will learn Go a hell of a lot quicker than Scala, IMO. A safer choice, for their team and position, IMO.
- oblio 13y ago"Big is beautiful" versus "small is beautiful". Some people like Ruby, others like Python. Hell, some people like Perl! :p
- stg 13y agoThis, whilst likely true, is not a self-evident fact. I think the argument in the blog post is weak because it does not quantify what makes Go more suitable for learning than say, Erlang or Clojure, which are dismissed outright, before moving on to discuss some of the advantages of Go in depth.
- txutxu 13y agoSomething is wrong with this statement: "Scala is great for performance, but quite bad at resource usage" Maybe it's directly related to the next statement: "No-one in the core GOV.UK team had a deep knowledge of Scala, and particularly how the old router worked" But the first statement, literally, does not make sense on its own.
- mseepgood 13y ago> But the first statement, literally, does not make sense on its own. Why? There are other resources than CPU time, e.g. memory.
- txutxu 13y ago"Computer performance is characterized by the amount of useful work accomplished by a computer system compared to the time and resources used." --wikipedia Is there a different definition I'm missing? I think, a car can be "fast" and take too much fuel, but (in my book) a programming language cannot be "great for performance" while being bad at resource usage.
- _pmf_ 13y ago> and particularly how the old router worked Oh, then it's OK. It's always reasonable and a safe bet to rewrite something instead of understanding how something works, because after all, your predecessors were a dumb bunch and you (using Go! how cool is that!?) obviously know much more.
- CmonDev 13y ago"Erlang, Go, Scala, and Clojure are all good fits and represent the current state of the art." - why is the choice so limited? It seems to be quite JVM-focused as well, I would consider F#/Nemerle as well. Erlang was invented in 1986 by the way - so a safe bet rather than "state of the art".
- danpalmer 13y agoF# and Nemerle are both based on the CLR, and unfortunately the performance of Mono is pretty bad, and not really suitable yet for high performance use-cases like this. Also I'd Erlang is state of the art, even though it is so old (Haskell too), but I wouldn't call it an entirely safe bet because it requires quite a different style of programming and experience, so might not work well for certain developers.
- CmonDev 13y agoIsn't Erlang just a (semi-)functional language with Actor Model baked in? I might be wrong but it seems that you just need any dev familiar with message queues and functional style.