8 ms·

Tackling Webdev as a Bioinformatician: why is it so hard?

139 points by LifeIsBio 6y ago

edoceo 6y ago
It's awesome to see an article get to the five why's. This is the Way.
- LifeIsBio 6y ago
  This has been my first experience with it and it seems like a great tool to ensure structured thinking. It's easy to give up a level or two deep and just say, "Oh, it's probably because..."
cpach 6y ago
Very thoughtful article. There is so much accidental complexity in software.
- LifeIsBio 6y ago
  It's kind of ironic. While I've been thinking about the complexity of software/webdev for a while, I was feeling a little unsure about posting this particular link because I wrote it in a hasty, semi-embittered fit of annoyance. Regardless, thanks for the read. :)
  - tejtm 6y ago
    hopefully this observation won't be taken too harshly... I have also worked in programming for the sciences my entire career, multiple decades, would not want to do anything else. There is an awful lot of actual complexity out there especially in the life sciences which is inherently messy ... a system of exceptions ... Sometimes looking at the stuff that has to happen now, just to give information away for free it is staggering. The only conclusion I come to is complexity envy, Because I can't believe all this inevitablely became necessary. Yes there are bad actors and good security is very hard, and much more so if you are transacting money, but in other cases you can't really steal what is being given away but now there is the expectation that all this extra stuff is par for the course it you are putting on the web.
MattGaiser 6y ago
Documentation is frighteningly bad across the industry and nobody has a good solution yet other than hoping that someone else will do something about it. And if you think Django is murky, try Spring :). Django is one of the easiest frameworks to work with simply because it includes so many functions you would otherwise need to somehow integrate. And even then there are still these issues. Good post.
- bayesian_horse 6y ago
  I think Django's documentation is fantastic. 90% of the time I have to look up something, google redirects me there anyway.
  - MattGaiser 6y ago
    Django's documentation is great for developers who know their way around documentation like that as it is very comprehensive. It drills down nicely and is quick and easy to understand if you otherwise know what else is going on. What it misses is mapping that documentation to the specific problem a person wants to solve or the feature they want to build. When I was learning Django a couple years ago I found the examples and best practices sparse.
    - jldugger 6y ago
      A long long time ago, there was a really great website called djangobook.com. It had like 20 chapters and projects explaining how things are done in Django, and each paragraph had a comments section where people could ask questions specific to that paragraph. That site probably did a lot for Django's success in the community. Unfortunately, giving away a free resource like that doesn't balance the books and years of neglect later the site no longer represented modern Django. It's entirely possible the new Django book site is just as good, but the paywall discourages novices and experts alike. My friends who stayed webdevs think Two Scoops of Django is close in scope, and comes with free updates. It's not quite the everbook that I had hoped the original site would be, but is highly focused on best practices.
bayesian_horse 6y ago
Trying to make the opposite transition, my quibble with web development is that much of the work seems so pointless. I hope that after this crisis we will care less about making people click ads than about learning how stuff works in biology.
- MattGaiser 6y ago
  We still run into the problem of how to pay for content on the web. Do we really want everything to either be volunteer or sponsored by someone?
  - Mati_Roy 6y ago
    it's a public good that will likely be underfunded if we stick with volunteers and sponsored except maybe if status is a sufficient payment? mayyybe good if altruistic ideas are more likely to propagate under those incentives?
- LifeIsBio 6y ago
  I heard there's a great new site that can help you learn about bio if you're interested... You know, if the site will let you login, that is. ;)
  - TheOtherHobbes 6y ago
    [Looks like you've read 0.1% of this page! Enter your email into this popup to receive emails you don't want about things you're not interested in. Also turn off your ad blocker, because we're not a charity, you freeloader.]
- Animats 6y ago
  Trying to make the opposite transition, my quibble with web development is that much of the work seems so pointless. That's the problem with web development today. Something that not only should be straightforward, but used to be straightforward, is now very complex. Partly because it involves connecting a large number of complex and constantly changing packages. The last time I wanted to do something with an API, I coded up the API in Go and ran it from FCGI to get multiple copies. Easier than all those "frameworks". Probably faster, too, since Go is hard-compiled.
juskrey 6y ago
Welcome to real world, academic Neo.
- LifeIsBio 6y ago
  As my PI used to say, "If it were easy, it wouldn't be fun. And we wouldn't have jobs."
CameronNemo 6y ago
looks like it got hugged https://web.archive.org/web/20200507060013/https://jessimekirk.com/blog/web_for_biofx/ https://web.archive.org/web/20200507060013/https://jessimeki...
- LifeIsBio 6y ago
  Oh no! It's github pages and totally static, so it should recover pretty easily, I hope. Thanks for the archive though!
  - saagarjha 6y ago
    Looks fine on my end. Good choice on using GitHub Pages by the way; had you not I’m sure half the comments would have been about how you would have it so much easier and moral it would be had you used that ;)
  - CameronNemo 6y ago
    Must have been a blip on my end then.
    - hutattedonmyarm 6y ago
      It definitely got hugged, I couldn't access it either for a bit. Recovered quickly enough though
saagarjha 6y ago
All programming has a barrier, but I think web development is a unique situation where there’s a lot of complexity because it’s so open to abuse. If you don’t do HTTPS right bad things can happen, so setting that up is hard. If you process payments bad things can happen, so setting that up is hard. If you send people email bad things can happen, so setting that up is hard. With things like game programming the worst that can happen is your game is slow or buggy or laggy but usually you’re not exposing it to the huge, often malicious audience. With web development you make one mistake and someone is stealing thousands of dollars from you and running cryptomining on your VPS while defacing your website.
- 29athrowaway 6y ago
  If you redirect users, bad things can happen. If you consume third party dependencies, bad things can happen. If you handle sensitive or personally identifiable information, bad things can happen. If you are not aware of privacy protection laws, bad things can happen. If you configure S3 buckets, bad things can happen. If you deploy static assets, better exclude sensitive information like version control data or bad things can happen. If you construct database queries, bad things can happen. If you manipulate files, bad things can happen. If your site can be put in an iframe, bad things can happen... and the list goes on and on.
- jamil7 6y ago
  Yeah and it results in so many small, seemingly insiginificant things that can bite you. Like the last point in which I guess the author had hardcoded their redis uri rather than using the environment variable provided by Heroku. How would anyone know about that unless they had been told by someone or been bitten in the past or had some understanding of how Heroku worked underneath. Edit: from the comments it's clear that OP didn't actually hardcode the connection string.
  - korijn 6y ago
    To be fair, haven't we all (eventually) learned to support environment configuration by messing it up in similar ways at some point?
ml_basics 6y ago
A great read!
KaiserPro 6y ago
I 100% feel your pain. I used to work for a very large financial news company. They were all javascript. I though to my self: I learnt JS in 2008, this should be easy (it was 2014). Apparently very hard. HTML isn't HTML anymore JS is forced everywhere, and everything has a different name. Anyway, I disappeared back into HPC/VFX/machinelearning, to re-appear into webdev in 2020. I was forced to do some front end work as part of a bootcamp. It took the best part of a week to make a bit of text a link. The template library was so far removed from anything googleable, there was no documentation, worse still no style guide or example style guide. I am glad I don't do web GUIs, they are hard and nasty.
- woutr_be 6y ago
  Honestly, they're only hard and nasty if you make them hard and nasty. A text link is as easy as <a href="">...</a>. You can of course use React with 2 different libraries to achieve the same goal, but why would you do that.
  - realharo 6y ago
    React by itself isn't difficult either. Anyone who has an hour or so to spare can go through the tutorial https://reactjs.org/tutorial/tutorial.html https://reactjs.org/tutorial/tutorial.html. They need to rewrite it to use hooks at some point, but it's still a pretty good intro.
    - moring 6y ago
      > They need to rewrite it to use hooks at some point What is the reason that not using hooks is so frowned upon? When I took my first steps with React, I didn't know hooks, so I write component classes and it immediately "clicked" with me because it behaves like most other GUI frameworks do, from Win32-based (1) to X-based to Android to iOS to macOS to Java, and tons of higher-level frameworks in between. It also feels obvious to write a component class because you are describing the behavior of a stateful thing, that is, exactly what classes are made for. With hooks, OTOH, we have a stateful "function", but all state is implicit, which describes behavior other than its main behavior (rendering) but that is implicit too... I can see why developers outside the React world don't "get" hooks, because I can't even remember seeing this idiom outside the React world. (BTW, I've seen the claim that hooks are easier to reason about, for which I found the opposite to be true, and the claim that this makes the main function stateless, which is absurd since the state just becomes implicit). (1) the raw system-level APIs are often not class-based, but nearly all frameworks above it are. edit: the footnote broke formatting
mrmonkeyman 6y ago
You are just new. These issues don't look complex at all. Welcome to engineering in general, not just software.
Hasu 6y ago
It's funny, when I was first learning programming 15 years ago, the thing I heard everywhere was "webdev is easy and for noobs", and sure, it was 15 years ago and webdev has come a long way... but when I interview the fresh-out-of-college candidates for junior positions, there's still the elitism of "If you aren't managing memory and flipping bits you aren't a REAL programmer." No, most web developers aren't implementing the most complex and intricate algorithms or 'close to the metal' or whatever other gatekeeping people want to claim, but that doesn't mean it's easy work. I've never had a job where I didn't see some green engineer laugh off a feature request with "Oh, that's easy, it's just calling an API and putting the information into some HTML, what's so hard about that?" and then getting stuck when their first implementation fails QA, or the product manager explains that the customer needs it to be ever so slightly different, and the complexity of those edge cases turn out to not be so easy to deal with as the world of perfect computer science might suggest. Is it hard in the same way? No, I will fully admit that coming up with a novel algorithmic solution requires a different mindset and brilliance that very few, if any, really good web developers I've met really have (although I'm certain there are people who excel at both, they're rare enough that I haven't been lucky enough to encounter them). But being really good at building a solid, secure, modern website isn't easy either, and it's also valuable. This post focused on the complexity of the tooling, which yes, is definitely a thing, but if you don't know web development, and you know that you don't know web development, why are you attempting to build something using complex high level tools designed for and used by advanced web developers? Then again, if you don't know web development and you don't know what you don't know, and you've just heard 'Django is good' how would you know to maybe pick Flask or something simpler to understand? I don't have an answer or a larger point other than this: writing software is difficult and complex and none of us should look down on anyone else because what they do seems easy on the outside.
- guggle 6y ago
  I wouldn't count Flask as simpler than Django. These 5 lines hello world examples are incredibly misleading.
  - eximius 6y ago
    Django is an enormous project that dwarfs Flask in size and complexity.
travolter 6y ago
Http is just clearly not made for the current state of the internet. It was obviously made for much simpler uses cases but has organically grown together with the internet. This has led to numerous hacks and workaroudns which often became the defacto standard to do things on the web. It's not necessarily an easy problem to solve, client server communication is hard. But http protocol seems like the least effective way to do it, but almost impossible to replace now.
- TheOtherHobbes 6y ago
  Applies even more so to CSS and JS. All the above are quick hacks that never really did the job properly in the first place. So they're propped up by a creaking pile of frameworks and other support systems that try to mitigate their awfulness, but mostly just add extra awfulness of their own. Meanwhile backend systems - core server and db infrastructure, user authentication, computing resource allocation, payments, media and content management, privacy, and project build systems - are even more byzantine and non-standardised and more or less have to be retooled from scratch for every project - a huge job which duplicates an incredible amount of work across the entire industry.
- tannhaeuser 6y ago
  > Http is just clearly not made for the current state of the internet As a corollary/reformulation of that sentence you get that what businesses are trying to do in the web is clearly something it wasn't made for. Yet businesses want to free-ride on the web's success, and are ruining it.
maelito 6y ago
> or the “right” framework to choose, changes from year to year I've been using React in all of my projets since it came out in 2013. That's 7 years. React's (+ node for backend if needed) been one of the recommended for at least 4 years. And lots of things have changed ! Using modules and building complex apps now with Webpack & co is so much easier than before. Not needing to support IE is also a blessing. The piece doesn't say how mobile dev would be easier. Or how embedded dev would be easier. It just says : dev is hard. Yes it is, it's a a full job. Hopefully nocde solutions will reduce the gap for some use cases.
- simongray 6y ago
  React - as a library, but especially as a UI paradigm - has clearly won the frontend war for now, taking the mantle from JQuery. I don't even use React directly, just its DOM renderer through a ClojureScript wrapper (reagent) and the high-level concepts are still basically the same. Still, when I glance at the JS frontend world people are still looking for the next hot frontend framework all the time, even if many are doing something similar to React. Going by github stars, Vue is the biggest one now. There's also Angular, Svelte, Ember, and Aurelia.
  - maelito 6y ago
    You're right. React and Vue are hard to compare though. The fact that React has no router, while react-router has 40k stars makes it a very imprecise metric.
tannhaeuser 6y ago
Is the author asking why putting up a paywall for Django hosted on Heroku with Stripe integration on what should be static content is difficult (for stories related to overcoming idiosyncrasies in using Python for bioinformatics, or all things)? I'd like to give a rebuttal, but the site (the Minimal Errors theme for Jekyll?) does click-jacking (blocks copy/paste) so I won't bother. I guess the answer is right there: webdev (and bioinformatics) is full of incidental complexity and zero-sum "solutions" advertised to you when the original purpose of the web is just publishing structured documents via HTML/SGML (even though I'd agree Python is great for learning, and Heroku has at least a reputation for being the easiest cloud hosting option out there).
- LifeIsBio 6y ago
  The site that's giving me problems isn't the site you were on, but another site I'm building: https://mycodestories.com/ https://mycodestories.com/ It's not static; it's a platform for helping people (life scientists) learn to code. In general, Heroku has been a great platform. The fact that you can just `git push`, and your dyno rebuilds and restarts and your changes are live is so cool.
bpizzi 6y ago
And it looks like you've just scratched the surface, you're not finished encountering walls yet. Hang in there, there's a relative important chance that the fog will clear for you in some months/years. The hard truth is, software engineering today is made of layers upon layers of abstractions. You have to dig threw those layers, understanding sufficiently the inherent and different complexity of each one before jumping to the next. And then finally it'll make sense, and new layers (there's a new layer every year btw) will be easier to grok. If it helps, we've all had to go threw this learning process. All of us. However I don't think everyone is made for enjoying this process, that's another hard truth. The real difficulty is sticking to the learning process while everything and everyone is screaming at you "Go! Build something! Ship Something! Show something you lazy b".
valera_rozuvan 6y ago
I start getting my friends up to speed by giving them just enough JavaScript + HTML to write a Tetris clone. Of course - I guide them along the way, answer any questions they have, and push them towards the right direction if they get stuck. If they can get through it - then they can make it in the world of WebDev, and software engineering in general. The main thing to remember is that you can only show the way - the person in question must actually step forwards and be motivated enough to go through the learning process. Otherwise - you need to think of another career. Programming is not for everyone IMHO.
- LifeIsBio 6y ago
  A little tangential, but learning to code through video games has always seemed like one of the best ways to do it, particularly with friends.
ertian 6y ago
I feel like interacting with there real world, and with users in particular, just inevitably leads to orders of magnitude more complexity. You can see that even in simple cases. Writing a basic single-propose script is trivial; it may end up being just a line or two. As soon as you decide to make it general and reusable, though, complexity explodes: positional arguments are confusing, better name them. Users are gonna expect a familiar format. Need a library for that. Reasonable and understandable names are tricky! What arguments are required versus optional? What if _this_ option is specified without _that_ one? Gotta add proper errors for every bad state! What if the user starts using globbing? Or wants to specify multiple inputs and one output, or vice versa? Your three-line script is now 100+ lines. And you're not even worried about hostile or abusive users yet! Writing code for the web is all that--and much more besides.
gexla 6y ago
You start with HTML. Write your HTML in an editor. You now have what we started with and I won't complain about your un-styled site. Right now I'm working on a simple personal thing with unstyled pages and a static site generator (Hugo.) The first Google search for hosting should be fine. There should be a simple tutorial for uploading your HTML file. That's all there is to it. Beyond that, maybe you're just trying to bite off more than you can chew because you think you need things. It would be like going to ACME engineering supply and grabbing a bunch of stuff off the shelf because it sounds good. Then you get home and you have no idea what these things do. A lot of this stuff looks deceptively like it all goes under the category of "web development." Sending an email isn't web development. That's dealing with a different protocol and different server software. You don't need to go far down that path before you start to wonder how any of this mess can work at all. I'm not even talking about web development, I feel this way all the way down to the metal. Yesterday I spent hours trying to troubleshoot an issue with Ryzen 3rd generation chips apparently having a problem when you load up all the slots with RAM (I threw in the towel because I can get by with half the RAM.) That has always been the case. Software hasn't gotten worse, it has always been bad. But functional. Maybe it has become more complex.
- Cu3PO42 6y ago
  Anecdotally I have a Ryzen 3600 on a B450 board with all RAM slots populated and have yet to notice an issue that isn't there when using only half the RAM.
  - gexla 6y ago
    Probably the wrong place for the discussion. Do a search and you'll find all sorts of people with this issue. I had the problem after I first built the computer. Then I was able to boot by lowering the speed of the memory in the BIOS, now I have the problem again after a reboot. I don't know what changed. There's hundreds of threads which cover a lot of ground on this issue. I'm just going to make this a long term project to understand the issue until I finally cave and buy something which has already been built. Edit: Also 3600 with B450 board. Kingston DDR4-2666 Ram (8x4) which is all the computer store here in smallish city in the Philippines had in stock.
goatinaboat 6y ago
Because it's artificially made hard by people who enjoy complexity for its own sake. The article is about implementing a shopping cart, as far as I can make out. Well people used to write those kinds of websites, that perfectly well enabled users to browse products, fill a basket, place the order and make a payment using nothing more than notepad.exe. Some software is legitimately hard to write, even some websites. But this isn't an example of that. The author has been sent on a wild goose chase.
- LifeIsBio 6y ago
  I'm curious where you got the idea that the site is a shopping cart? CodeStories is a site to help life scientists learn to program by giving them problems, then letting them submit scripts and then auto-graded their scripts for correctness.
  - goatinaboat 6y ago
    CodeStories is a site to help life scientists learn to program by giving them problems, then letting them submit scripts and then auto-graded their scripts for correctness. Because it’s a pretty generic e-commerce application that could have been built on the web 25 years ago. It should not have been complicated to do.
mattlondon 6y ago
Tackling bioinformatics as a webdev: why is it so hard? "I am a professional in Field X but I am doing work in Field Y despite having minimal skills or experience and I ran into loads of problems and issues that I don't know how to fix! Field Y is total bullshit and people need to make it easier so people with no skills or experience can do it! Wah wah!" ... or to be less facetious, modern web/frontend development is a genuine skill set that benefits from experience and knowledge. Should we expect anyone to just jump in cold with minimal experience and not find it difficult? E.g. in the same way would we expect people to start bioinformatics cold and be productive immediately? I'd doubt it. On the past web dev has been looked down upon as somehow "lesser" than "real" engineering, with sneering criticism from people more used to traditional development approaches (particularly criticism of JavaScript). It is just some web pages right? HTML and CSS? How hard can it be? Well guess what: it is a genuine skillset and people should not expect to just be able to jump into something without knowledge training or experience and be proficient at it, just like we'd not for brain surgery, car mechanics, concert piano, structural engineering, or indeed bioinformatics. </rant>
- hmwhy 6y ago
  I second this. I have some experience in both web development and scientific computing in various fields (not bioinformatics). Things felt just as hard at the beginning in all cases—I don't recall things being any better in scientific computing compared to web development. I agree that documentation is terrible most of the time, but that's not something limited to web development in my own experience. (On the contrary, check out the documentation of the website in question in this thread: https://news.ycombinator.com/item?id=23031408 https://news.ycombinator.com/item?id=23031408, it's absolutely fantastic). Some things become second nature when you have developed expertise in that field, and it's easy to criticise something that you have comparatively less knowledge in and pass it off as "too difficult", particularly when the web development part seems to be a means to an end (which seems to be the case here). Edit: missing conjunction, tidied up the last sentence for clarity.
  - LifeIsBio 6y ago
    Oh, interesting. I just wrote a reply saying the exact opposite. I remember learning scientific computing as being much less frustrating. Sometimes you'd have some error spawning out of some ancient C/Fortran library. But that was rare for me. Even after a couple of years on the web, I still feel like I encounter issues where I just don't even know how to start.
jldugger 6y ago
> Why’s there so much data in the url? A common refrain I’ll use for the remainder of the post is, “I don’t know”. One of my biggest gripes with web programming is the number of rabbit holes to fall down. There’s just not enough time. I mean, bioinformatics is _notorious_ for spending just enough time working on software to publish a result and no more. If we want to talk about rabbitholes I nominate BioPerl. > here are a list of terms that you’d never come across as a bioinformatician: ["X-Forwarded-Proto", "proxying", "requests", "TLS connection", "HTTP", "loopback/localhost", "whitelisted proxy servers", "gunicorn", "nginx", "Apache/httpd", "HAProxy", "Django", "SECURE_SSL_REDIRECT", "SecurityMiddleware", "adapter", "URI"] This list is more than a little disingenuous: Django: This is like, the framework OP built their software on top of. I don't know how one gets the point of filing this bug without knowing the term Django requests: literally, a user's web request. HTTP/URI: this is the web developer equivalent of a biologist not knowing the term DNA. Or DOI for that matter. whitelist: the opposite of a blacklist Much of rest comes down to the simple fact that job of web developers is to build a network service. This is primarily done with layers; each layer only needs to know how to interface with the adjoining layers above and below, like a lego brick. This is actually a _solution_ to the complexity, allowing us to constrain the design of each layer to interact only with those above and below it. This layer abstraction lets us reason about each layer locally, independently of the system as a whole. nginx, Apache/httpd, HAProxy: in this context, mentioned as possible proxies forming the outermost layer. in this context, accepts HTTP/HTTPS connections and makes HTTP connections to gunicorn gunicorn: the thing that runs the Django app. Accepts HTTP connections, makes WSGI calls to the Django app Django: Accepts WSGI function calls, routes them through middleware, then to the app as you define in urlpatterns SecurityMiddleware: a class that potentially modifies inbound requests and outbound responses Finally, OP is in the unhappy situation of bridging two network protocols: HTTP and email. The normal trick of using a protocol relative URI will not work here because you've gone outside the browser request-response loop. This puts them in the unhappy state of the lowest layer needing to know the details of the highest layer, hence the discussion of absolute uris. This is a direct violation of the local reasoning constraint, and what makes all of this hard. X-Forwarded-Proto: An HTTP header, added to incoming HTTP requests designed to indicate to layers below which URI scheme was originally used, HTTP or HTTPS. SECURE_SSL_REDIRECT: a setting for SecurityMiddleware that will stop processing the request and send a response to the browser telling it to resend the query over https (an HTTP 302 redirect), if the incoming connection is seen as coming over not http. tl;dr: email is the thing that sucks.
teekert 6y ago
In a related article: "Tackling bioinformatics as a Webdev: Why is it so hard?" Maybe Webdev is just complex? But tbh, as a bioinformatician (in Next Generation Sequencing mainly) I actually find Webdev easier. I mean, we face Bam files, Sam files, Cram files and annotate them with Bed files (with many different poorly documented standards) or GTF files depending on your Ensembl or UCSC references persuasion... Some file types are 0-based inclusive, some are 1 based exclusive, some call chromosome just ints (1,2, and then error when X, Y if you really assume they are ints) some call them chr1, chr2, chrX etc. We have fastq files with different quality encodings depending on their age, I recently had some strange errors, turns out the R2 of a fastq file is reverse complementary from the standard (why?!)... It's a mess and many people that spend years in the field still don't know more than just "choose one side and stick with it"... We also have a huge amount of frameworks for data analysis pipelines, Snakemake, Bpipe, Broad's WDL, etc, etc, etc. Not to mention the insance amounts of read mappers (BWA, RSEM, Tophat 1 and 2, Kallisto etc). Need I go on?
- LifeIsBio 6y ago
  That's super interesting. If someone asked me if bioinformatics was complex, I'd say "Yes, massively so." But then, I would take it in a totally different direction. To me, the complexity of bioinformatics come from the biology. How do you think about weird biological edge cases? What do your distributions look like and how do you choose thresholds? How do you even form a good question? Thinks like [B|S|Cr]am files are complex in their own way, but mostly they're just kinda annoying to have to figure out. You can dig in as deep as you need pretty easily. On the other hand, maybe I'm over-simplifying because I feel more comfortable in this domain, which I think is your point.
  - teekert 6y ago
    Well, I come from Biology, always had Webdev as a hobby and then moved into bioinformatics which now turns towards software development. So Biology feels most familiar, but it is indeed also always complex. Btw, the files are ok and with effort can be understood and maybe some could even be considered elegant. It's just the shear amount of "standards" or rather the lack thereof.
rgoulter 6y ago
I don't think the author is saying "why front end application development hard?" so much as "why is setting up a website hard?". On one end of the spectrum, simply putting content on a website isn't so hard, right? On the other end, making a web application is understandably difficult. Features like users, payment forms, and email confirmation (so e.g. the website can be used for a course) sounds like they ought to be as simple as the former, but are closer to the latter. Is the value proposition from the website building websites like Wix that they take care of the hard stuff?
namelosw 6y ago
It is no brainer for professionals, but to be frank, it's too hard for everyone. It's time for a new PHP.
deleted 6y ago
[deleted]
IshKebab 6y ago
Backwards compatibility. It's possible to design coherent easy to use UI systems. Flutter and Qt are great and both much easier to develop for than the web IMO. But they don't have to worry about running in web browsers.
sradman 6y ago
I love these kinds of retrospectives as they give insight into the learning process. We tend to focus on the acquisition of specialized knowledge but there is often a body of shared knowledge, the context you might call it, that we pick up indirectly. You have to read between the lines but what we have in this post is a Data Scientist skilled in the Python programming language struggling to acquire the shared knowledge required to build and deploy Web Apps. One toolchain (Python), one framework (Django), one backend architecture (12-Factor Apps) but the narrative surrounding URLs, HTTP, DNS, TLS, REST APIs, JSON, CORS, and www-forms is missing. We seem to be missing a simple way to learn the shared knowledge needed to build apps in a world reshaped by Web, Mobile, and Cloud technologies.
- LifeIsBio 6y ago
  I’m not sure exactly what it is about this comment, but I really like it. “Context” is a great word for this idea. There are a bunch of small/medium complexity things that need to play well together to get things to work, and it can feel trickier than necessary.
not_a_moth 6y ago
This is common among academics, who expect full transparency and understanding in what they're doing, instead of a "get it working and go" attitude. The problem is the OP didn't test anything: He launched and users couldn't log in. Ideally I'd like to gain 100% visibility into everything related to my tasks in webdev, but it's impractical and counter productive.
deleted 6y ago
[deleted]
dade_ 6y ago
It is hard to get the hang of Django is because the polls tutorial isn't helpful to actually understand Django and no one wrote this book yet, but I think Nigel has done a great job: https://djangobook.com/mastering-django-2-book/ https://djangobook.com/mastering-django-2-book/
heisnotanalien 6y ago
Because instead of fixing genuine problems and making it easier people invent new languages and frameworks to do things in their own way.
cmrdporcupine 6y ago
Honestly I've always found UI development or anything with some aspect of UI in general has the potential to devolve into a tangled mess. I've avoided web facing development for the last 10 years of my career, and have transitioned to embedded work. Every time I traipse through job postings and look at the tech stack companies are using ("wanted: full stack developer") I just shake my head. I am sure most of the choices are justified, but it all seems so brutally overwrought from the outside looking in. But beyond a fetish for novelty in our industry, I think that's just endemic to user facing development.
walleeee 6y ago
Related: institutions and labs need to recognize that 'research software engineer' is an increasingly necessary role and attract/fund more of them.
thrower123 6y ago
Yes, web development is hard and stupidly complicated and slapdash and we all hate it. It sucks, and the whole thing should be burned to the ground, the ashes sown with salt, and pissed on by donkeys. Unfortunately we're stuck with this Rube Goldberg collection of excrement, and it pays the bills. So there's nothing much you can do except lean in and embrace the suck. Maybe hit yourself in the forehead a few times with a clawhammer or start carrying a hipflask to keep yourself at happy-drunk.
6510 6y ago
People who understand the mess are much more influential and they are moving the show forwards. Making things easy for newcomers is just not very high on the agenda (anymore) Perhaps things have to get complicated enough for it to get priority, perhaps cool things will happen all of a sudden. I was highly amused by the spreadsheet programming tool a while back. It seems a giant leap in the other direction.

139 points by LifeIsBio 6y ago

edoceo 6y ago

It's awesome to see an article get to the five why's. This is the Way.

LifeIsBio 6y ago
This has been my first experience with it and it seems like a great tool to ensure structured thinking. It's easy to give up a level or two deep and just say, "Oh, it's probably because..."

cpach 6y ago

Very thoughtful article. There is so much accidental complexity in software.

LifeIsBio 6y ago
It's kind of ironic. While I've been thinking about the complexity of software/webdev for a while, I was feeling a little unsure about posting this particular link because I wrote it in a hasty, semi-embittered fit of annoyance. Regardless, thanks for the read. :)
- tejtm 6y ago
  hopefully this observation won't be taken too harshly... I have also worked in programming for the sciences my entire career, multiple decades, would not want to do anything else. There is an awful lot of actual complexity out there especially in the life sciences which is inherently messy ... a system of exceptions ... Sometimes looking at the stuff that has to happen now, just to give information away for free it is staggering. The only conclusion I come to is complexity envy, Because I can't believe all this inevitablely became necessary. Yes there are bad actors and good security is very hard, and much more so if you are transacting money, but in other cases you can't really steal what is being given away but now there is the expectation that all this extra stuff is par for the course it you are putting on the web.

MattGaiser 6y ago

Documentation is frighteningly bad across the industry and nobody has a good solution yet other than hoping that someone else will do something about it. And if you think Django is murky, try Spring :). Django is one of the easiest frameworks to work with simply because it includes so many functions you would otherwise need to somehow integrate. And even then there are still these issues. Good post.

bayesian_horse 6y ago
I think Django's documentation is fantastic. 90% of the time I have to look up something, google redirects me there anyway.
- MattGaiser 6y ago
  Django's documentation is great for developers who know their way around documentation like that as it is very comprehensive. It drills down nicely and is quick and easy to understand if you otherwise know what else is going on. What it misses is mapping that documentation to the specific problem a person wants to solve or the feature they want to build. When I was learning Django a couple years ago I found the examples and best practices sparse.
  - jldugger 6y ago
    A long long time ago, there was a really great website called djangobook.com. It had like 20 chapters and projects explaining how things are done in Django, and each paragraph had a comments section where people could ask questions specific to that paragraph. That site probably did a lot for Django's success in the community. Unfortunately, giving away a free resource like that doesn't balance the books and years of neglect later the site no longer represented modern Django. It's entirely possible the new Django book site is just as good, but the paywall discourages novices and experts alike. My friends who stayed webdevs think Two Scoops of Django is close in scope, and comes with free updates. It's not quite the everbook that I had hoped the original site would be, but is highly focused on best practices.

bayesian_horse 6y ago

Trying to make the opposite transition, my quibble with web development is that much of the work seems so pointless. I hope that after this crisis we will care less about making people click ads than about learning how stuff works in biology.

MattGaiser 6y ago
We still run into the problem of how to pay for content on the web. Do we really want everything to either be volunteer or sponsored by someone?
- Mati_Roy 6y ago
  it's a public good that will likely be underfunded if we stick with volunteers and sponsored except maybe if status is a sufficient payment? mayyybe good if altruistic ideas are more likely to propagate under those incentives?
LifeIsBio 6y ago
I heard there's a great new site that can help you learn about bio if you're interested... You know, if the site will let you login, that is. ;)
- TheOtherHobbes 6y ago
  [Looks like you've read 0.1% of this page! Enter your email into this popup to receive emails you don't want about things you're not interested in. Also turn off your ad blocker, because we're not a charity, you freeloader.]
Animats 6y ago
Trying to make the opposite transition, my quibble with web development is that much of the work seems so pointless. That's the problem with web development today. Something that not only should be straightforward, but used to be straightforward, is now very complex. Partly because it involves connecting a large number of complex and constantly changing packages. The last time I wanted to do something with an API, I coded up the API in Go and ran it from FCGI to get multiple copies. Easier than all those "frameworks". Probably faster, too, since Go is hard-compiled.

juskrey 6y ago

Welcome to real world, academic Neo.

LifeIsBio 6y ago
As my PI used to say, "If it were easy, it wouldn't be fun. And we wouldn't have jobs."

CameronNemo 6y ago

looks like it got hugged https://web.archive.org/web/20200507060013/https://jessimekirk.com/blog/web_for_biofx/ https://web.archive.org/web/20200507060013/https://jessimeki...

LifeIsBio 6y ago
Oh no! It's github pages and totally static, so it should recover pretty easily, I hope. Thanks for the archive though!
- saagarjha 6y ago
  Looks fine on my end. Good choice on using GitHub Pages by the way; had you not I’m sure half the comments would have been about how you would have it so much easier and moral it would be had you used that ;)
- CameronNemo 6y ago
  Must have been a blip on my end then.
  - hutattedonmyarm 6y ago
    It definitely got hugged, I couldn't access it either for a bit. Recovered quickly enough though

saagarjha 6y ago

All programming has a barrier, but I think web development is a unique situation where there’s a lot of complexity because it’s so open to abuse. If you don’t do HTTPS right bad things can happen, so setting that up is hard. If you process payments bad things can happen, so setting that up is hard. If you send people email bad things can happen, so setting that up is hard. With things like game programming the worst that can happen is your game is slow or buggy or laggy but usually you’re not exposing it to the huge, often malicious audience. With web development you make one mistake and someone is stealing thousands of dollars from you and running cryptomining on your VPS while defacing your website.

29athrowaway 6y ago
If you redirect users, bad things can happen. If you consume third party dependencies, bad things can happen. If you handle sensitive or personally identifiable information, bad things can happen. If you are not aware of privacy protection laws, bad things can happen. If you configure S3 buckets, bad things can happen. If you deploy static assets, better exclude sensitive information like version control data or bad things can happen. If you construct database queries, bad things can happen. If you manipulate files, bad things can happen. If your site can be put in an iframe, bad things can happen... and the list goes on and on.
jamil7 6y ago
Yeah and it results in so many small, seemingly insiginificant things that can bite you. Like the last point in which I guess the author had hardcoded their redis uri rather than using the environment variable provided by Heroku. How would anyone know about that unless they had been told by someone or been bitten in the past or had some understanding of how Heroku worked underneath. Edit: from the comments it's clear that OP didn't actually hardcode the connection string.
- korijn 6y ago
  To be fair, haven't we all (eventually) learned to support environment configuration by messing it up in similar ways at some point?

ml_basics 6y ago

A great read!

KaiserPro 6y ago

I 100% feel your pain. I used to work for a very large financial news company. They were all javascript. I though to my self: I learnt JS in 2008, this should be easy (it was 2014). Apparently very hard. HTML isn't HTML anymore JS is forced everywhere, and everything has a different name. Anyway, I disappeared back into HPC/VFX/machinelearning, to re-appear into webdev in 2020. I was forced to do some front end work as part of a bootcamp. It took the best part of a week to make a bit of text a link. The template library was so far removed from anything googleable, there was no documentation, worse still no style guide or example style guide. I am glad I don't do web GUIs, they are hard and nasty.

woutr_be 6y ago
Honestly, they're only hard and nasty if you make them hard and nasty. A text link is as easy as <a href="">...</a>. You can of course use React with 2 different libraries to achieve the same goal, but why would you do that.
- realharo 6y ago
  React by itself isn't difficult either. Anyone who has an hour or so to spare can go through the tutorial https://reactjs.org/tutorial/tutorial.html https://reactjs.org/tutorial/tutorial.html. They need to rewrite it to use hooks at some point, but it's still a pretty good intro.
  - moring 6y ago
    > They need to rewrite it to use hooks at some point What is the reason that not using hooks is so frowned upon? When I took my first steps with React, I didn't know hooks, so I write component classes and it immediately "clicked" with me because it behaves like most other GUI frameworks do, from Win32-based (1) to X-based to Android to iOS to macOS to Java, and tons of higher-level frameworks in between. It also feels obvious to write a component class because you are describing the behavior of a stateful thing, that is, exactly what classes are made for. With hooks, OTOH, we have a stateful "function", but all state is implicit, which describes behavior other than its main behavior (rendering) but that is implicit too... I can see why developers outside the React world don't "get" hooks, because I can't even remember seeing this idiom outside the React world. (BTW, I've seen the claim that hooks are easier to reason about, for which I found the opposite to be true, and the claim that this makes the main function stateless, which is absurd since the state just becomes implicit). (1) the raw system-level APIs are often not class-based, but nearly all frameworks above it are. edit: the footnote broke formatting

mrmonkeyman 6y ago

You are just new. These issues don't look complex at all. Welcome to engineering in general, not just software.

Hasu 6y ago

It's funny, when I was first learning programming 15 years ago, the thing I heard everywhere was "webdev is easy and for noobs", and sure, it was 15 years ago and webdev has come a long way... but when I interview the fresh-out-of-college candidates for junior positions, there's still the elitism of "If you aren't managing memory and flipping bits you aren't a REAL programmer." No, most web developers aren't implementing the most complex and intricate algorithms or 'close to the metal' or whatever other gatekeeping people want to claim, but that doesn't mean it's easy work. I've never had a job where I didn't see some green engineer laugh off a feature request with "Oh, that's easy, it's just calling an API and putting the information into some HTML, what's so hard about that?" and then getting stuck when their first implementation fails QA, or the product manager explains that the customer needs it to be ever so slightly different, and the complexity of those edge cases turn out to not be so easy to deal with as the world of perfect computer science might suggest. Is it hard in the same way? No, I will fully admit that coming up with a novel algorithmic solution requires a different mindset and brilliance that very few, if any, really good web developers I've met really have (although I'm certain there are people who excel at both, they're rare enough that I haven't been lucky enough to encounter them). But being really good at building a solid, secure, modern website isn't easy either, and it's also valuable. This post focused on the complexity of the tooling, which yes, is definitely a thing, but if you don't know web development, and you know that you don't know web development, why are you attempting to build something using complex high level tools designed for and used by advanced web developers? Then again, if you don't know web development and you don't know what you don't know, and you've just heard 'Django is good' how would you know to maybe pick Flask or something simpler to understand? I don't have an answer or a larger point other than this: writing software is difficult and complex and none of us should look down on anyone else because what they do seems easy on the outside.

guggle 6y ago
I wouldn't count Flask as simpler than Django. These 5 lines hello world examples are incredibly misleading.
- eximius 6y ago
  Django is an enormous project that dwarfs Flask in size and complexity.

travolter 6y ago

Http is just clearly not made for the current state of the internet. It was obviously made for much simpler uses cases but has organically grown together with the internet. This has led to numerous hacks and workaroudns which often became the defacto standard to do things on the web. It's not necessarily an easy problem to solve, client server communication is hard. But http protocol seems like the least effective way to do it, but almost impossible to replace now.

TheOtherHobbes 6y ago
Applies even more so to CSS and JS. All the above are quick hacks that never really did the job properly in the first place. So they're propped up by a creaking pile of frameworks and other support systems that try to mitigate their awfulness, but mostly just add extra awfulness of their own. Meanwhile backend systems - core server and db infrastructure, user authentication, computing resource allocation, payments, media and content management, privacy, and project build systems - are even more byzantine and non-standardised and more or less have to be retooled from scratch for every project - a huge job which duplicates an incredible amount of work across the entire industry.
tannhaeuser 6y ago
> Http is just clearly not made for the current state of the internet As a corollary/reformulation of that sentence you get that what businesses are trying to do in the web is clearly something it wasn't made for. Yet businesses want to free-ride on the web's success, and are ruining it.

maelito 6y ago

> or the “right” framework to choose, changes from year to year I've been using React in all of my projets since it came out in 2013. That's 7 years. React's (+ node for backend if needed) been one of the recommended for at least 4 years. And lots of things have changed ! Using modules and building complex apps now with Webpack & co is so much easier than before. Not needing to support IE is also a blessing. The piece doesn't say how mobile dev would be easier. Or how embedded dev would be easier. It just says : dev is hard. Yes it is, it's a a full job. Hopefully nocde solutions will reduce the gap for some use cases.

simongray 6y ago
React - as a library, but especially as a UI paradigm - has clearly won the frontend war for now, taking the mantle from JQuery. I don't even use React directly, just its DOM renderer through a ClojureScript wrapper (reagent) and the high-level concepts are still basically the same. Still, when I glance at the JS frontend world people are still looking for the next hot frontend framework all the time, even if many are doing something similar to React. Going by github stars, Vue is the biggest one now. There's also Angular, Svelte, Ember, and Aurelia.
- maelito 6y ago
  You're right. React and Vue are hard to compare though. The fact that React has no router, while react-router has 40k stars makes it a very imprecise metric.

tannhaeuser 6y ago

Is the author asking why putting up a paywall for Django hosted on Heroku with Stripe integration on what should be static content is difficult (for stories related to overcoming idiosyncrasies in using Python for bioinformatics, or all things)? I'd like to give a rebuttal, but the site (the Minimal Errors theme for Jekyll?) does click-jacking (blocks copy/paste) so I won't bother. I guess the answer is right there: webdev (and bioinformatics) is full of incidental complexity and zero-sum "solutions" advertised to you when the original purpose of the web is just publishing structured documents via HTML/SGML (even though I'd agree Python is great for learning, and Heroku has at least a reputation for being the easiest cloud hosting option out there).

LifeIsBio 6y ago
The site that's giving me problems isn't the site you were on, but another site I'm building: https://mycodestories.com/ https://mycodestories.com/ It's not static; it's a platform for helping people (life scientists) learn to code. In general, Heroku has been a great platform. The fact that you can just `git push`, and your dyno rebuilds and restarts and your changes are live is so cool.

bpizzi 6y ago

And it looks like you've just scratched the surface, you're not finished encountering walls yet. Hang in there, there's a relative important chance that the fog will clear for you in some months/years. The hard truth is, software engineering today is made of layers upon layers of abstractions. You have to dig threw those layers, understanding sufficiently the inherent and different complexity of each one before jumping to the next. And then finally it'll make sense, and new layers (there's a new layer every year btw) will be easier to grok. If it helps, we've all had to go threw this learning process. All of us. However I don't think everyone is made for enjoying this process, that's another hard truth. The real difficulty is sticking to the learning process while everything and everyone is screaming at you "Go! Build something! Ship Something! Show something you lazy b".

valera_rozuvan 6y ago

I start getting my friends up to speed by giving them just enough JavaScript + HTML to write a Tetris clone. Of course - I guide them along the way, answer any questions they have, and push them towards the right direction if they get stuck. If they can get through it - then they can make it in the world of WebDev, and software engineering in general. The main thing to remember is that you can only show the way - the person in question must actually step forwards and be motivated enough to go through the learning process. Otherwise - you need to think of another career. Programming is not for everyone IMHO.

LifeIsBio 6y ago
A little tangential, but learning to code through video games has always seemed like one of the best ways to do it, particularly with friends.

ertian 6y ago

I feel like interacting with there real world, and with users in particular, just inevitably leads to orders of magnitude more complexity. You can see that even in simple cases. Writing a basic single-propose script is trivial; it may end up being just a line or two. As soon as you decide to make it general and reusable, though, complexity explodes: positional arguments are confusing, better name them. Users are gonna expect a familiar format. Need a library for that. Reasonable and understandable names are tricky! What arguments are required versus optional? What if _this_ option is specified without _that_ one? Gotta add proper errors for every bad state! What if the user starts using globbing? Or wants to specify multiple inputs and one output, or vice versa? Your three-line script is now 100+ lines. And you're not even worried about hostile or abusive users yet! Writing code for the web is all that--and much more besides.

gexla 6y ago

You start with HTML. Write your HTML in an editor. You now have what we started with and I won't complain about your un-styled site. Right now I'm working on a simple personal thing with unstyled pages and a static site generator (Hugo.) The first Google search for hosting should be fine. There should be a simple tutorial for uploading your HTML file. That's all there is to it. Beyond that, maybe you're just trying to bite off more than you can chew because you think you need things. It would be like going to ACME engineering supply and grabbing a bunch of stuff off the shelf because it sounds good. Then you get home and you have no idea what these things do. A lot of this stuff looks deceptively like it all goes under the category of "web development." Sending an email isn't web development. That's dealing with a different protocol and different server software. You don't need to go far down that path before you start to wonder how any of this mess can work at all. I'm not even talking about web development, I feel this way all the way down to the metal. Yesterday I spent hours trying to troubleshoot an issue with Ryzen 3rd generation chips apparently having a problem when you load up all the slots with RAM (I threw in the towel because I can get by with half the RAM.) That has always been the case. Software hasn't gotten worse, it has always been bad. But functional. Maybe it has become more complex.

Cu3PO42 6y ago
Anecdotally I have a Ryzen 3600 on a B450 board with all RAM slots populated and have yet to notice an issue that isn't there when using only half the RAM.
- gexla 6y ago
  Probably the wrong place for the discussion. Do a search and you'll find all sorts of people with this issue. I had the problem after I first built the computer. Then I was able to boot by lowering the speed of the memory in the BIOS, now I have the problem again after a reboot. I don't know what changed. There's hundreds of threads which cover a lot of ground on this issue. I'm just going to make this a long term project to understand the issue until I finally cave and buy something which has already been built. Edit: Also 3600 with B450 board. Kingston DDR4-2666 Ram (8x4) which is all the computer store here in smallish city in the Philippines had in stock.

goatinaboat 6y ago

Because it's artificially made hard by people who enjoy complexity for its own sake. The article is about implementing a shopping cart, as far as I can make out. Well people used to write those kinds of websites, that perfectly well enabled users to browse products, fill a basket, place the order and make a payment using nothing more than notepad.exe. Some software is legitimately hard to write, even some websites. But this isn't an example of that. The author has been sent on a wild goose chase.

LifeIsBio 6y ago
I'm curious where you got the idea that the site is a shopping cart? CodeStories is a site to help life scientists learn to program by giving them problems, then letting them submit scripts and then auto-graded their scripts for correctness.
- goatinaboat 6y ago
  CodeStories is a site to help life scientists learn to program by giving them problems, then letting them submit scripts and then auto-graded their scripts for correctness. Because it’s a pretty generic e-commerce application that could have been built on the web 25 years ago. It should not have been complicated to do.

mattlondon 6y ago

Tackling bioinformatics as a webdev: why is it so hard? "I am a professional in Field X but I am doing work in Field Y despite having minimal skills or experience and I ran into loads of problems and issues that I don't know how to fix! Field Y is total bullshit and people need to make it easier so people with no skills or experience can do it! Wah wah!" ... or to be less facetious, modern web/frontend development is a genuine skill set that benefits from experience and knowledge. Should we expect anyone to just jump in cold with minimal experience and not find it difficult? E.g. in the same way would we expect people to start bioinformatics cold and be productive immediately? I'd doubt it. On the past web dev has been looked down upon as somehow "lesser" than "real" engineering, with sneering criticism from people more used to traditional development approaches (particularly criticism of JavaScript). It is just some web pages right? HTML and CSS? How hard can it be? Well guess what: it is a genuine skillset and people should not expect to just be able to jump into something without knowledge training or experience and be proficient at it, just like we'd not for brain surgery, car mechanics, concert piano, structural engineering, or indeed bioinformatics. </rant>

hmwhy 6y ago
I second this. I have some experience in both web development and scientific computing in various fields (not bioinformatics). Things felt just as hard at the beginning in all cases—I don't recall things being any better in scientific computing compared to web development. I agree that documentation is terrible most of the time, but that's not something limited to web development in my own experience. (On the contrary, check out the documentation of the website in question in this thread: https://news.ycombinator.com/item?id=23031408 https://news.ycombinator.com/item?id=23031408, it's absolutely fantastic). Some things become second nature when you have developed expertise in that field, and it's easy to criticise something that you have comparatively less knowledge in and pass it off as "too difficult", particularly when the web development part seems to be a means to an end (which seems to be the case here). Edit: missing conjunction, tidied up the last sentence for clarity.
- LifeIsBio 6y ago
  Oh, interesting. I just wrote a reply saying the exact opposite. I remember learning scientific computing as being much less frustrating. Sometimes you'd have some error spawning out of some ancient C/Fortran library. But that was rare for me. Even after a couple of years on the web, I still feel like I encounter issues where I just don't even know how to start.

jldugger 6y ago

> Why’s there so much data in the url? A common refrain I’ll use for the remainder of the post is, “I don’t know”. One of my biggest gripes with web programming is the number of rabbit holes to fall down. There’s just not enough time. I mean, bioinformatics is _notorious_ for spending just enough time working on software to publish a result and no more. If we want to talk about rabbitholes I nominate BioPerl. > here are a list of terms that you’d never come across as a bioinformatician: ["X-Forwarded-Proto", "proxying", "requests", "TLS connection", "HTTP", "loopback/localhost", "whitelisted proxy servers", "gunicorn", "nginx", "Apache/httpd", "HAProxy", "Django", "SECURE_SSL_REDIRECT", "SecurityMiddleware", "adapter", "URI"] This list is more than a little disingenuous: Django: This is like, the framework OP built their software on top of. I don't know how one gets the point of filing this bug without knowing the term Django requests: literally, a user's web request. HTTP/URI: this is the web developer equivalent of a biologist not knowing the term DNA. Or DOI for that matter. whitelist: the opposite of a blacklist Much of rest comes down to the simple fact that job of web developers is to build a network service. This is primarily done with layers; each layer only needs to know how to interface with the adjoining layers above and below, like a lego brick. This is actually a _solution_ to the complexity, allowing us to constrain the design of each layer to interact only with those above and below it. This layer abstraction lets us reason about each layer locally, independently of the system as a whole. nginx, Apache/httpd, HAProxy: in this context, mentioned as possible proxies forming the outermost layer. in this context, accepts HTTP/HTTPS connections and makes HTTP connections to gunicorn gunicorn: the thing that runs the Django app. Accepts HTTP connections, makes WSGI calls to the Django app Django: Accepts WSGI function calls, routes them through middleware, then to the app as you define in urlpatterns SecurityMiddleware: a class that potentially modifies inbound requests and outbound responses Finally, OP is in the unhappy situation of bridging two network protocols: HTTP and email. The normal trick of using a protocol relative URI will not work here because you've gone outside the browser request-response loop. This puts them in the unhappy state of the lowest layer needing to know the details of the highest layer, hence the discussion of absolute uris. This is a direct violation of the local reasoning constraint, and what makes all of this hard. X-Forwarded-Proto: An HTTP header, added to incoming HTTP requests designed to indicate to layers below which URI scheme was originally used, HTTP or HTTPS. SECURE_SSL_REDIRECT: a setting for SecurityMiddleware that will stop processing the request and send a response to the browser telling it to resend the query over https (an HTTP 302 redirect), if the incoming connection is seen as coming over not http. tl;dr: email is the thing that sucks.

teekert 6y ago

In a related article: "Tackling bioinformatics as a Webdev: Why is it so hard?" Maybe Webdev is just complex? But tbh, as a bioinformatician (in Next Generation Sequencing mainly) I actually find Webdev easier. I mean, we face Bam files, Sam files, Cram files and annotate them with Bed files (with many different poorly documented standards) or GTF files depending on your Ensembl or UCSC references persuasion... Some file types are 0-based inclusive, some are 1 based exclusive, some call chromosome just ints (1,2, and then error when X, Y if you really assume they are ints) some call them chr1, chr2, chrX etc. We have fastq files with different quality encodings depending on their age, I recently had some strange errors, turns out the R2 of a fastq file is reverse complementary from the standard (why?!)... It's a mess and many people that spend years in the field still don't know more than just "choose one side and stick with it"... We also have a huge amount of frameworks for data analysis pipelines, Snakemake, Bpipe, Broad's WDL, etc, etc, etc. Not to mention the insance amounts of read mappers (BWA, RSEM, Tophat 1 and 2, Kallisto etc). Need I go on?

LifeIsBio 6y ago
That's super interesting. If someone asked me if bioinformatics was complex, I'd say "Yes, massively so." But then, I would take it in a totally different direction. To me, the complexity of bioinformatics come from the biology. How do you think about weird biological edge cases? What do your distributions look like and how do you choose thresholds? How do you even form a good question? Thinks like [B|S|Cr]am files are complex in their own way, but mostly they're just kinda annoying to have to figure out. You can dig in as deep as you need pretty easily. On the other hand, maybe I'm over-simplifying because I feel more comfortable in this domain, which I think is your point.
- teekert 6y ago
  Well, I come from Biology, always had Webdev as a hobby and then moved into bioinformatics which now turns towards software development. So Biology feels most familiar, but it is indeed also always complex. Btw, the files are ok and with effort can be understood and maybe some could even be considered elegant. It's just the shear amount of "standards" or rather the lack thereof.

rgoulter 6y ago

I don't think the author is saying "why front end application development hard?" so much as "why is setting up a website hard?". On one end of the spectrum, simply putting content on a website isn't so hard, right? On the other end, making a web application is understandably difficult. Features like users, payment forms, and email confirmation (so e.g. the website can be used for a course) sounds like they ought to be as simple as the former, but are closer to the latter. Is the value proposition from the website building websites like Wix that they take care of the hard stuff?

namelosw 6y ago

It is no brainer for professionals, but to be frank, it's too hard for everyone. It's time for a new PHP.

deleted 6y ago

[deleted]

IshKebab 6y ago

Backwards compatibility. It's possible to design coherent easy to use UI systems. Flutter and Qt are great and both much easier to develop for than the web IMO. But they don't have to worry about running in web browsers.

sradman 6y ago

I love these kinds of retrospectives as they give insight into the learning process. We tend to focus on the acquisition of specialized knowledge but there is often a body of shared knowledge, the context you might call it, that we pick up indirectly. You have to read between the lines but what we have in this post is a Data Scientist skilled in the Python programming language struggling to acquire the shared knowledge required to build and deploy Web Apps. One toolchain (Python), one framework (Django), one backend architecture (12-Factor Apps) but the narrative surrounding URLs, HTTP, DNS, TLS, REST APIs, JSON, CORS, and www-forms is missing. We seem to be missing a simple way to learn the shared knowledge needed to build apps in a world reshaped by Web, Mobile, and Cloud technologies.

LifeIsBio 6y ago
I’m not sure exactly what it is about this comment, but I really like it. “Context” is a great word for this idea. There are a bunch of small/medium complexity things that need to play well together to get things to work, and it can feel trickier than necessary.

not_a_moth 6y ago

This is common among academics, who expect full transparency and understanding in what they're doing, instead of a "get it working and go" attitude. The problem is the OP didn't test anything: He launched and users couldn't log in. Ideally I'd like to gain 100% visibility into everything related to my tasks in webdev, but it's impractical and counter productive.

deleted 6y ago

[deleted]

dade_ 6y ago

It is hard to get the hang of Django is because the polls tutorial isn't helpful to actually understand Django and no one wrote this book yet, but I think Nigel has done a great job: https://djangobook.com/mastering-django-2-book/ https://djangobook.com/mastering-django-2-book/

heisnotanalien 6y ago

Because instead of fixing genuine problems and making it easier people invent new languages and frameworks to do things in their own way.

cmrdporcupine 6y ago

Honestly I've always found UI development or anything with some aspect of UI in general has the potential to devolve into a tangled mess. I've avoided web facing development for the last 10 years of my career, and have transitioned to embedded work. Every time I traipse through job postings and look at the tech stack companies are using ("wanted: full stack developer") I just shake my head. I am sure most of the choices are justified, but it all seems so brutally overwrought from the outside looking in. But beyond a fetish for novelty in our industry, I think that's just endemic to user facing development.

walleeee 6y ago

Related: institutions and labs need to recognize that 'research software engineer' is an increasingly necessary role and attract/fund more of them.

thrower123 6y ago

Yes, web development is hard and stupidly complicated and slapdash and we all hate it. It sucks, and the whole thing should be burned to the ground, the ashes sown with salt, and pissed on by donkeys. Unfortunately we're stuck with this Rube Goldberg collection of excrement, and it pays the bills. So there's nothing much you can do except lean in and embrace the suck. Maybe hit yourself in the forehead a few times with a clawhammer or start carrying a hipflask to keep yourself at happy-drunk.

6510 6y ago

People who understand the mess are much more influential and they are moving the show forwards. Making things easy for newcomers is just not very high on the agenda (anymore) Perhaps things have to get complicated enough for it to get priority, perhaps cool things will happen all of a sudden. I was highly amused by the spreadsheet programming tool a while back. It seems a giant leap in the other direction.