9 ms·

Boring Python: Code quality

228 points by masenf 4y ago

toastal 4y ago
Not agreeing/disagreeing with the message, but the style of writing here is quite nice. It's focused, reasoned, and doesn't make too many assumptions about your tools and environment--and I appreciate that acknowledgment.
leetrout 4y ago
> If your project builds a Docker container, also create a .dockerignore file to specify files and directories that should be excluded from the container. I would nitpick this. You build images not containers and since files are not copied by default there is more nuance here that the .dockerignore file makes builds faster by not including them in the build context. That does ultimately prevent COPY directives from using them but it is these sorts of brief, slightly inaccurate summaries that mislead folks as they build understanding.
- nigamanth 4y ago
  Shouldn't the speeding up of the build make the program less boring? From my understanding, the program gets more boring as the time it takes an application to build increases. > slightly inaccurate Not entirely, I'm not sure the author even wanted to stress on this in the article. People won't learn docker from a python article about the same.
_8j50 4y ago
I don't work on large python projects, mostly just small scripts that need to work well (integrating with a 3rd party rest api is a good example). I don't do CI or unittests but I use git. This is because it takes time and honestly no one outside of myself would care for small stuff like that. But I do run autopep8 and pylint it (I ignore stuff like line being too long,broad exception handling or lack of docs). My concern is a) It needs to be reliable (don't wanna spend a ton of time chasing bugs later on) b) How can I write the actual code better? I see what pro devs write and they use smarter language features or better organization of the code itself that makes it faster and reliable, I wish I could learn that explicitly somewhere. I mean, just the 2.7->3.0 jump was big for me because since I don't code regularly that meant googling errors a lot basically. Even now, I dread new python versions because some dependency would start using those features and that means I have to use venv to get that small script to work and then figure out how to troubleshoot bugs in that other lib's code with the new feature so I can do a PR for them. I love python but this is exactly why I prioritize languages that don't churn out new drastic features quickly. Those are just not suitable for people whose day job is not coding and migrating to new versions, supporting code bases, messing with build systems, unit tests, qa,ci,etc... coding is a tool for me, not the centerpiece of all I do. But python is still great despite all that.
- _bohm 4y ago
  For your dependency/versioning issue, use a virtualenv per-project and pin your dependency versions in requirements.txt
  - LtWorf 4y ago
    Don't pin unless it's needed. I have a library… most downloaded version is 3 years old. The newer versions are massively faster but nobody uses them.
    - anderskaseorg 4y ago
      You should not pin the public requirements that get uploaded with a library (listed in setup.py, setup.cfg, or pyproject.toml), since that will restrict your downstream users, leading to version conflicts and persistent security vulnerabilities. But it’s totally reasonable to pin the private requirements that you develop it against (listed in requirements.txt, poetry.lock, or similar), updating them every so often during the course of development, so that contributors can use a consistent set of tools.
      - LtWorf 4y ago
        So leaf packages can pin vulnerable or slow stuff why?
        anderskaseorg 4y ago
        The context is > For your dependency/versioning issue, use a virtualenv per-project and pin your dependency versions in requirements.txt requirements.txt is not uploaded to PyPI and has no effect on your package’s dependencies when a user installs it (leaf package or no). It’s only used for developing the package itself, typically in a unique virtual environment.
    - _bohm 4y ago
      Agree in principle, but I'm giving advice to someone who programs on occasion and is primarily concerned with their programs breaking due to dependency version upgrades when they come back to them after a little while.
bombolo 4y ago
> I recommend using two tools together: Black and isort. Black formats things differently depending on the version. So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth. isort's completely random… For example the latest version I tried decided to alphabetically sort all the imports, regardless if they are part of standard library or 3rd party. This is a big change of behaviour from what it was doing before. All those big changes introduce commits that make git bisect generally slower. Which might be awful if you also have some C code to recompile at every step of bisecting.
- zzzeek 4y ago
  > Black formats things differently depending on the version. So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth. use pre-commit https://pre-commit.com/ https://pre-commit.com/ so that everyone is on the same version for commits.
- kuu 4y ago
  > Black formats things differently depending on the version. Then add black as part of your environment with an specific version...
  - bombolo 4y ago
    Or wait until a more sensible formatting tool comes along. Reformatting the whole code every version isn't so good. It's also very slow.
    - fnord123 4y ago
      Install pre-commit: https://pre-commit.com/ https://pre-commit.com/ Set black up in the pre-commit with a specific version. When you make a commit it will black the files being committed using the specific version of black. As it's a subset, it's fast. As it's a specific version, it's not going back and forth. I hope this solves your issues.
anderskaseorg 4y ago
If you aren’t happy with Flake8, Pylint, and isort (or maybe if you are!), I recommend checking out Ruff: https://github.com/charliermarsh/ruff https://github.com/charliermarsh/ruff It’s literally 100 times faster, with comparable coverage to Flake8 plus dozens of plugins, automatic fixes, and very active development.
- gjulianm 4y ago
  Does ruff replace isort? Because I'm really unhappy with it, it doesn't work with tabs and conflicts with yapf all the time.
  - anderskaseorg 4y ago
    Yes, as of last month. I’m not sure if it works with YAPF; it’s designed to work with Black and doesn’t currently have many of isort’s configuration options. Worth a try!
    - gjulianm 4y ago
      Thanks! The configuration options don't matter too much, I was really unhappy with the options isort gives.
  - drcongo 4y ago
    > it doesn't work with tabs What do you mean by this? Are you indenting Python with tabs?
    - makapuf 4y ago
      If your import statements are indented, they must be in control statements (try/except or conditional imports, I fail to see why you would put those in a loop) thus they will be difficult to reorder if you import another module (with different name or stdlib status) on import error.
      - 4y ago
LarsDu88 4y ago
I disagree with this assessment on running a static type checker, although I will admit, every update of python over the past 3 years seems to add more and more typing changes which tends to force global typing updates (looking at you Numpy for python 3.12!) When python converges on consistent typing across its extended numpy and pandas ecosystem, I believe we will be able to move towards a fully JIT'd language.
- bombolo 4y ago
  > I believe we will be able to move towards a fully JIT'd language. Unless they actually go ahead with the deferred evaluation of types (PEP 563), make all types strings at runtime and make it impossible to know which type they actually are. :) But they will probably not: https://discuss.python.org/t/type-annotations-pep-649-and-pep-563/11363 https://discuss.python.org/t/type-annotations-pep-649-and-pe... But it could be a breaking change in the language. As it is, I can run this "a: str = 3" and it will work.
tilschuenemann 4y ago
There is also a 'hypermodern' cookie cutter template for python projects - I've used it several times now and it works mostly out of the box: https://github.com/cjolowicz/cookiecutter-hypermodern-python https://github.com/cjolowicz/cookiecutter-hypermodern-python
- jmduke 4y ago
  I love this template as well, and wholeheartedly recommend it. There are a couple things you probably don't need (click and nox, for instance, seem only useful if you're really building a couple specific things) but the gestalt of it is really strong. The [article series](https://medium.com/@cjolowicz/hypermodern-python-d44485d9d769 https://medium.com/@cjolowicz/hypermodern-python-d44485d9d76...) that spawned the template is worth reading in full. I would go so far as to say that the hypermodern template, nomenclature aside, is strictly better than the recommendations that the OP put forward both here and in the previous essay on dependency management. Poetry and ruff, for instance, are both very good tools — and I can understand _not_ recommending them for one reason or another but to not even mention them strikes me as worrisome.
bvrmn 4y ago
> Coverage measurements are too easy to “game” — you can get to 100% coverage without meaningfully testing all or even most of your code Still it's a good low bar for testing. It's easy and rises code quality. I have very good results with coverage driving colleagues to write tests. And on code review we can discuss how to make tests more useful and robust and how to decrease number of mocks, etc.
- tasuki 4y ago
  Hard disagree: 100% coverage is not a "good low bar" and does not increase code quality. Depending on the language and the particular project, my sweet spot for test coverage is between 30-70%, testing the tricky bits. I've seen 100% code coverage with tests for all the getters and setters. These tests were not only 100% useless, they actively hindered any changes to the system.
  - Lutger 4y ago
    This is true. You can have bad unittests which make the system worse and you would be better of without them. You can also have useless unittests with 100% coverage, which is pretty much the same as bad tests because more code means more bugs and more work. Unittests are also code after all. The only thing you can say about a very low coverage is that you probably don't have good tests. That's not a very useful metric, since you likely already know that. The metric 'coverage' is almost useless. Code coverage starts to be useful once you let go of it as a goal and ignore the total percentage number. I found it is very useful though if you can generate detailed reports on each line of code or better yet, each branch in the code, indicating whether that line or branch is tested. Eyeball all the lines which don't have tests and ask yourself: would it be useful to add a test exercising this codepath? How do I make sure it works and what cases can I think of that could go wrong? This doesn't automatically lead to good tests, but it helps you spot where you should focus your testing efforts. Code coverage is a good tool to help think of test cases, as a metric for the total codebase it is nearly useless.
    - hbrn 4y ago
      > Code coverage starts to be useful once you let go of it as a goal and ignore the total percentage number When a measure becomes a target, it ceases to be a good measure. It takes immense discipline to actually let go of a metric to keep it valuable.
liendolucas 4y ago
I don't understand. The title of the post is: "Boring Python: code quality". Further down: "Today I want to talk about what's generally called "code quality" - tools to help...". I'm sorry but "code quality" is not "tooling". The post should be titled: "Python tooling". Code quality: What abstractions are you using in your code?, How easy is to make a change?, How easy is to understand your code base?, What patterns are you using and why?, Are you abusing class inheritance?, How many side effects are present out there and how does that affect your program?, Are you taking advantage of the Python language facilities and idioms?, Is it easy to write unit tests for?, etc. To sum up: "tooling" != "code quality".
- simonw 4y ago
  "Boring Python" is the title of the series of posts, which started here: https://www.b-list.org/weblog/2022/may/13/boring-python-dependencies/ https://www.b-list.org/weblog/2022/may/13/boring-python-depe... > This is the first in hopefully a series of posts I intend to write about how to build/manage/deploy/etc. Python applications in as boring a way as possible. It's a riff on Boring Technology, see https://boringtechnology.club/ https://boringtechnology.club/
  - liendolucas 4y ago
    It doesn't really matter if it is fun, sad, entertaining or boring Python. The post wrongly claims that putting all these tools in a project will lead to "code quality". It says that at the very beginning as I quoted it. This is harmful, especially for a junior developer or someone that doesn't have much or none experience coding. It will make the naive reader believe that having those tools in place quality code is being produced.
    - Cthulhu_ 4y ago
      While it doesn't produce quality, it can help a developer write better code; I've learned tons about JS internals just by sticking to whatever eslint popped up with. Before that I learned tons about Java's internals and best practices by fixing issues that these automated "best practice" tools came up with. It's far from perfect, but it helps if you don't know any better. And most people don't know any better. I'm currently spinning up a new project (React Native, Typescript) and I'm spending a lot of effort in locking down the project - eslint, unit tests & coverage, CI, strict typescript rules, etc - because this did not happen with the previous iteration of this project, leading to tens of thousands of LOC worth of unit- and end-to-end integration tests to become worthless and unusable. Sure, that was a lack of developer discipline as well, but why rely on other people when you can do it through technology as well? You can't control everyone.
anderskaseorg 4y ago
> For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it. This is an odd complaint. typing.Sequence[T] has been there since the first iteration of typing (3.5), for exactly that use case, along with many related collection types. https://docs.python.org/3/library/typing.html https://docs.python.org/3/library/typing.html mypy isn’t perfect, but it’s sure better than making things up without any checks; you’re going to want it for all but the smallest projects.
- ReflectedImage 4y ago
  You should never be using static typing with a scripting language like Python or Ruby. Dynamically typed code is 1/3rd the size of statically typed code, that means that one developer who is using dynamic typing is equivalent to 3 developers using statically typed code via MyPy. Since the code is 1/3rd of the size it contains 1/3rd of the bugs. This is confirmed by all the studies that have been done on the topic. If you use a static type checking with Python, you have increased your development time by 3 and your bug count by 3. Static typing's advantage is that the code runs a lot faster but that's only true if the language itself is statically typed. So with Python you have just screwed up.
  - bombolo 4y ago
    > Dynamically typed code is 1/3rd the size of statically typed code, This is absolutely not true. > Since the code is 1/3rd of the size it contains 1/3rd of the bugs. That is made up and contrary to all empirical evidence I've ever collected. I'd be curious if you have a source, but I doubt it.
    - ReflectedImage 4y ago
      Anyone with experience of writing both dynamic typed and statically typed can tell you that. Infact, you could just try it out for yourself. But here is your internet source for this blatantly obvious fact: https://games.greggman.com/game/dynamic-typing-static-typing/ https://games.greggman.com/game/dynamic-typing-static-typing...
c120 4y ago
Not sure if I like the recommendation to not let Black change your code and just give out errors. I absolutely let Black change code and see the value in Black that it does that so the devs do not have to spend time on manually formatting code. Black shouldn't break anything (and hasn't broken anything for me in the years I used it) but in the unlikely case it does it, there's still pytests/unittests after that that should catch problems...
- bsdz 4y ago
  IIRC, Black also checks byte code before and after formatting to ensure source code functionality is unaffected.
  - progval 4y ago
    Black checks the AST, not the byte code.
- dustrider 4y ago
  As I understood it, it was to not let black do the formatting during CI builds. In local dev you’d let it reformat. Even while it won’t break anything you want CI to be your safety net, flagging a local setup as being wrong is more valuable than magically autocorrecting it.
  - nine_k 4y ago
    We have Black as a pre-commit hook; works fine, even if it disagrees with your IDE a little bit sometimes. CI/CD has no business changing your code; it builds stuff using it, exactly as if commit such-and-such.
    - 411111111111111 4y ago
      > CI/CD has no business changing your code; it builds stuff using it, exactly as if commit such-and-such. That going too far unless you define code to be a subset of the files checked into the repository and simply define any file that's touched in an automated manner to be not code There are a lot of useful automations that can be part of the CI/CD pipeline, such as increasing a version number, generating a changelog, creating new deployment configuration etc They don't have to be part of it and it's possible to work around it/don't commit... But that comes with it's own challenges and issues
aitchnyu 4y ago
I wish VSCode would figure out that ExampleModel.objects.first() returns ExampleModel or None or ExampleModel.objects.filter() returns an iterable of ExampleModel. Has anybody gotten this working, automatically or manually annotating?
- jsmeaton 4y ago
  You can annotate the manager and get some typing help in the editor. And there’s django-stubs which helps a little when running mypy. It’s not as good as pycharm though. https://github.com/typeddjango/django-stubs/tree/master https://github.com/typeddjango/django-stubs/tree/master
  - aitchnyu 4y ago
    Could you share a guide on that?
    - jsmeaton 4y ago
      I don't have anything specific but here's something I quickly threw together demonstrating what I mean https://gist.github.com/jarshwah/1e683416d2ed2df28f254fc787d0d304 https://gist.github.com/jarshwah/1e683416d2ed2df28f254fc787d...
- jerrygenser 4y ago
  It's not a shortcoming of vscode it's due to the dynamic untyped nature of Django models unless you have a plugin or add typing to your own managers
mau 4y ago
One thing that is underestimated is keep the tools version in sync between your app dev dependencies and pre-commit. This also includes plugins for specific tools (for instance flake8). A solution would be to define the hooks in pre-commit to run the tools inside your venv. About typings: I agree the eco-system is not mature enough, especially for some frameworks such as Django, but the effort is still valuable and in many cases the static analysis provided by mypy is more useful than not using it at all. So I would suggest to try do your best to make it work.
modeopfer 4y ago
What's the current state of the art of managing multiple virtual environments, running tests and running your application? On Ubuntu and Windows I use Poetry [0], and it works, although it has (had?) some quirks during the installation on Windows. I liked its portability and lockfile format though. A few years ago I used conda [1], which was nice because it came batteries included especially for Deep Learning stuff. I switched because it felt way to heavy for porting scripts and small applications to constrained devices like a Raspberry Pi. And then there are also Docker Images, which I use if I want to give an application to somebody that "just works". What's your method of choice? [0] https://python-poetry.org/ https://python-poetry.org/ [1] https://www.anaconda.com/ https://www.anaconda.com/
- wil421 4y ago
  >I switched because it felt way to heavy for porting scripts and small applications to constrained devices like a Raspberry Pi. Agreed. I like docker images for smallish portable scripts. At home I can develop on my Mac and port it to a Raspberry PI or another x86 Windows/Linux box. Planning on running a docker swarm with a few Pi’s to see how it works.
- gjulianm 4y ago
  I use pip-tools to build a requirements.txt file from a requirements.in file. It does basically the same as poetry, but more manually. For me that's good because one of the application has a lot of requirements, and it needs to be deployed on systems with different Python versions, and the requirements need to be packaged along with the application because the servers have very limited internet access. So as long as Poetry doesn't add good support for multiple python versions and/or easy packaging of all dependencies, it isn't worth it for me to do the migration.
- Flex247A 4y ago
  If you feel that Anaconda is too heavy, try Miniconda [0]. The base environment is a standard Python 3.9 environment without any additional packages. [0] https://docs.conda.io/en/latest/miniconda.html https://docs.conda.io/en/latest/miniconda.html
jerrygenser 4y ago
Even since the start of python typing, it was recommended to use a more generic type like Iterable instead of List. The author claims that List is too specific -- this seems like a straw man argument against typing that doesn't acknowledge python's own advice. Also, mypy has gotten really good in recent years and I can vouch that on projects that have typing I catch bugs much much sooner. Previously I would only catch bugs when unit testing, now they are much more commonly type errors. The other thing typing does is allow for refactoring code. If anything, high code quality relates to the ability to refactor code confidently and typing helps this. Therefore I would put it at the top of the list above all the tooling presented (exception I agree with ci/cd)
- whalesalad 4y ago
  > Even since the start of python typing, it was recommended to use a more generic type like Iterable instead of List. The author claims that List is too specific These statements contradict themselves? List is too specific, and Sequence[item] is preferred. Sometimes you are dealing with a tuple, or a generator, and so it makes more sense to annotate that it is a generic iterable versus a concrete list.
  - jerrygenser 4y ago
    From the original article: > For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it. Yet the Python type-annotation ecosystem was strongly oriented around nominal typing (i.e., caring that something is exactly a list) from the beginning. I'm saying that this quote is a straw man and that contrary to what is claimed in the quote, instead, the ecosystem would go with/recommend Iterable[Item] or Sequence[Item] and not List[Item] if applicable. I think we both agree, not sure which part of my comment you think is contradictory.
  - hbrn 4y ago
    Whether something is generic/specific depends on the context. As an argument type, Iterable is permissive (generic). As a return type, Iterable is restrictive (specific).
IshKebab 4y ago
> For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it. Terrible advice not to use type hints and this reason makes no sense. There's already pretty good support for Sequence and Iterable and so on, and if you run into a place where you really can't write down the types (e.g. kwargs, which a lot of Python programmers abuse), then you can use Any. Blows my mind how allergic Python programmers are to static typing despite the huge and obvious benefits. It's true that Python's static typing does suck balls compared to most languages, but they're still a gazillion times better than nothing, and most of the reason they suck so much is that so many Python developers don't use them!

228 points by masenf 4y ago

Not agreeing/disagreeing with the message, but the style of writing here is quite nice. It's focused, reasoned, and doesn't make too many assumptions about your tools and environment--and I appreciate that acknowledgment.

leetrout 4y ago

> If your project builds a Docker container, also create a .dockerignore file to specify files and directories that should be excluded from the container. I would nitpick this. You build images not containers and since files are not copied by default there is more nuance here that the .dockerignore file makes builds faster by not including them in the build context. That does ultimately prevent COPY directives from using them but it is these sorts of brief, slightly inaccurate summaries that mislead folks as they build understanding.

nigamanth 4y ago
Shouldn't the speeding up of the build make the program less boring? From my understanding, the program gets more boring as the time it takes an application to build increases. > slightly inaccurate Not entirely, I'm not sure the author even wanted to stress on this in the article. People won't learn docker from a python article about the same.

_8j50 4y ago

I don't work on large python projects, mostly just small scripts that need to work well (integrating with a 3rd party rest api is a good example). I don't do CI or unittests but I use git. This is because it takes time and honestly no one outside of myself would care for small stuff like that. But I do run autopep8 and pylint it (I ignore stuff like line being too long,broad exception handling or lack of docs). My concern is a) It needs to be reliable (don't wanna spend a ton of time chasing bugs later on) b) How can I write the actual code better? I see what pro devs write and they use smarter language features or better organization of the code itself that makes it faster and reliable, I wish I could learn that explicitly somewhere. I mean, just the 2.7->3.0 jump was big for me because since I don't code regularly that meant googling errors a lot basically. Even now, I dread new python versions because some dependency would start using those features and that means I have to use venv to get that small script to work and then figure out how to troubleshoot bugs in that other lib's code with the new feature so I can do a PR for them. I love python but this is exactly why I prioritize languages that don't churn out new drastic features quickly. Those are just not suitable for people whose day job is not coding and migrating to new versions, supporting code bases, messing with build systems, unit tests, qa,ci,etc... coding is a tool for me, not the centerpiece of all I do. But python is still great despite all that.

_bohm 4y ago
For your dependency/versioning issue, use a virtualenv per-project and pin your dependency versions in requirements.txt
- LtWorf 4y ago
  Don't pin unless it's needed. I have a library… most downloaded version is 3 years old. The newer versions are massively faster but nobody uses them.
  - anderskaseorg 4y ago
    You should not pin the public requirements that get uploaded with a library (listed in setup.py, setup.cfg, or pyproject.toml), since that will restrict your downstream users, leading to version conflicts and persistent security vulnerabilities. But it’s totally reasonable to pin the private requirements that you develop it against (listed in requirements.txt, poetry.lock, or similar), updating them every so often during the course of development, so that contributors can use a consistent set of tools.
    - LtWorf 4y ago
      So leaf packages can pin vulnerable or slow stuff why?
      - anderskaseorg 4y ago
        The context is > For your dependency/versioning issue, use a virtualenv per-project and pin your dependency versions in requirements.txt requirements.txt is not uploaded to PyPI and has no effect on your package’s dependencies when a user installs it (leaf package or no). It’s only used for developing the package itself, typically in a unique virtual environment.
  - _bohm 4y ago
    Agree in principle, but I'm giving advice to someone who programs on occasion and is primarily concerned with their programs breaking due to dependency version upgrades when they come back to them after a little while.

bombolo 4y ago

> I recommend using two tools together: Black and isort. Black formats things differently depending on the version. So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth. isort's completely random… For example the latest version I tried decided to alphabetically sort all the imports, regardless if they are part of standard library or 3rd party. This is a big change of behaviour from what it was doing before. All those big changes introduce commits that make git bisect generally slower. Which might be awful if you also have some C code to recompile at every step of bisecting.

zzzeek 4y ago
> Black formats things differently depending on the version. So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth. use pre-commit https://pre-commit.com/ https://pre-commit.com/ so that everyone is on the same version for commits.
kuu 4y ago
> Black formats things differently depending on the version. Then add black as part of your environment with an specific version...
- bombolo 4y ago
  Or wait until a more sensible formatting tool comes along. Reformatting the whole code every version isn't so good. It's also very slow.
  - fnord123 4y ago
    Install pre-commit: https://pre-commit.com/ https://pre-commit.com/ Set black up in the pre-commit with a specific version. When you make a commit it will black the files being committed using the specific version of black. As it's a subset, it's fast. As it's a specific version, it's not going back and forth. I hope this solves your issues.

anderskaseorg 4y ago

If you aren’t happy with Flake8, Pylint, and isort (or maybe if you are!), I recommend checking out Ruff: https://github.com/charliermarsh/ruff https://github.com/charliermarsh/ruff It’s literally 100 times faster, with comparable coverage to Flake8 plus dozens of plugins, automatic fixes, and very active development.

gjulianm 4y ago
Does ruff replace isort? Because I'm really unhappy with it, it doesn't work with tabs and conflicts with yapf all the time.
- anderskaseorg 4y ago
  Yes, as of last month. I’m not sure if it works with YAPF; it’s designed to work with Black and doesn’t currently have many of isort’s configuration options. Worth a try!
  - gjulianm 4y ago
    Thanks! The configuration options don't matter too much, I was really unhappy with the options isort gives.
- drcongo 4y ago
  > it doesn't work with tabs What do you mean by this? Are you indenting Python with tabs?
  - makapuf 4y ago
    If your import statements are indented, they must be in control statements (try/except or conditional imports, I fail to see why you would put those in a loop) thus they will be difficult to reorder if you import another module (with different name or stdlib status) on import error.
    - 4y ago

LarsDu88 4y ago

I disagree with this assessment on running a static type checker, although I will admit, every update of python over the past 3 years seems to add more and more typing changes which tends to force global typing updates (looking at you Numpy for python 3.12!) When python converges on consistent typing across its extended numpy and pandas ecosystem, I believe we will be able to move towards a fully JIT'd language.

bombolo 4y ago
> I believe we will be able to move towards a fully JIT'd language. Unless they actually go ahead with the deferred evaluation of types (PEP 563), make all types strings at runtime and make it impossible to know which type they actually are. :) But they will probably not: https://discuss.python.org/t/type-annotations-pep-649-and-pep-563/11363 https://discuss.python.org/t/type-annotations-pep-649-and-pe... But it could be a breaking change in the language. As it is, I can run this "a: str = 3" and it will work.

tilschuenemann 4y ago

There is also a 'hypermodern' cookie cutter template for python projects - I've used it several times now and it works mostly out of the box: https://github.com/cjolowicz/cookiecutter-hypermodern-python https://github.com/cjolowicz/cookiecutter-hypermodern-python

jmduke 4y ago
I love this template as well, and wholeheartedly recommend it. There are a couple things you probably don't need (click and nox, for instance, seem only useful if you're really building a couple specific things) but the gestalt of it is really strong. The [article series](https://medium.com/@cjolowicz/hypermodern-python-d44485d9d769 https://medium.com/@cjolowicz/hypermodern-python-d44485d9d76...) that spawned the template is worth reading in full. I would go so far as to say that the hypermodern template, nomenclature aside, is strictly better than the recommendations that the OP put forward both here and in the previous essay on dependency management. Poetry and ruff, for instance, are both very good tools — and I can understand _not_ recommending them for one reason or another but to not even mention them strikes me as worrisome.

bvrmn 4y ago

> Coverage measurements are too easy to “game” — you can get to 100% coverage without meaningfully testing all or even most of your code Still it's a good low bar for testing. It's easy and rises code quality. I have very good results with coverage driving colleagues to write tests. And on code review we can discuss how to make tests more useful and robust and how to decrease number of mocks, etc.

tasuki 4y ago
Hard disagree: 100% coverage is not a "good low bar" and does not increase code quality. Depending on the language and the particular project, my sweet spot for test coverage is between 30-70%, testing the tricky bits. I've seen 100% code coverage with tests for all the getters and setters. These tests were not only 100% useless, they actively hindered any changes to the system.
- Lutger 4y ago
  This is true. You can have bad unittests which make the system worse and you would be better of without them. You can also have useless unittests with 100% coverage, which is pretty much the same as bad tests because more code means more bugs and more work. Unittests are also code after all. The only thing you can say about a very low coverage is that you probably don't have good tests. That's not a very useful metric, since you likely already know that. The metric 'coverage' is almost useless. Code coverage starts to be useful once you let go of it as a goal and ignore the total percentage number. I found it is very useful though if you can generate detailed reports on each line of code or better yet, each branch in the code, indicating whether that line or branch is tested. Eyeball all the lines which don't have tests and ask yourself: would it be useful to add a test exercising this codepath? How do I make sure it works and what cases can I think of that could go wrong? This doesn't automatically lead to good tests, but it helps you spot where you should focus your testing efforts. Code coverage is a good tool to help think of test cases, as a metric for the total codebase it is nearly useless.
  - hbrn 4y ago
    > Code coverage starts to be useful once you let go of it as a goal and ignore the total percentage number When a measure becomes a target, it ceases to be a good measure. It takes immense discipline to actually let go of a metric to keep it valuable.

liendolucas 4y ago

I don't understand. The title of the post is: "Boring Python: code quality". Further down: "Today I want to talk about what's generally called "code quality" - tools to help...". I'm sorry but "code quality" is not "tooling". The post should be titled: "Python tooling". Code quality: What abstractions are you using in your code?, How easy is to make a change?, How easy is to understand your code base?, What patterns are you using and why?, Are you abusing class inheritance?, How many side effects are present out there and how does that affect your program?, Are you taking advantage of the Python language facilities and idioms?, Is it easy to write unit tests for?, etc. To sum up: "tooling" != "code quality".

simonw 4y ago
"Boring Python" is the title of the series of posts, which started here: https://www.b-list.org/weblog/2022/may/13/boring-python-dependencies/ https://www.b-list.org/weblog/2022/may/13/boring-python-depe... > This is the first in hopefully a series of posts I intend to write about how to build/manage/deploy/etc. Python applications in as boring a way as possible. It's a riff on Boring Technology, see https://boringtechnology.club/ https://boringtechnology.club/
- liendolucas 4y ago
  It doesn't really matter if it is fun, sad, entertaining or boring Python. The post wrongly claims that putting all these tools in a project will lead to "code quality". It says that at the very beginning as I quoted it. This is harmful, especially for a junior developer or someone that doesn't have much or none experience coding. It will make the naive reader believe that having those tools in place quality code is being produced.
  - Cthulhu_ 4y ago
    While it doesn't produce quality, it can help a developer write better code; I've learned tons about JS internals just by sticking to whatever eslint popped up with. Before that I learned tons about Java's internals and best practices by fixing issues that these automated "best practice" tools came up with. It's far from perfect, but it helps if you don't know any better. And most people don't know any better. I'm currently spinning up a new project (React Native, Typescript) and I'm spending a lot of effort in locking down the project - eslint, unit tests & coverage, CI, strict typescript rules, etc - because this did not happen with the previous iteration of this project, leading to tens of thousands of LOC worth of unit- and end-to-end integration tests to become worthless and unusable. Sure, that was a lack of developer discipline as well, but why rely on other people when you can do it through technology as well? You can't control everyone.

anderskaseorg 4y ago

> For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it. This is an odd complaint. typing.Sequence[T] has been there since the first iteration of typing (3.5), for exactly that use case, along with many related collection types. https://docs.python.org/3/library/typing.html https://docs.python.org/3/library/typing.html mypy isn’t perfect, but it’s sure better than making things up without any checks; you’re going to want it for all but the smallest projects.

ReflectedImage 4y ago
You should never be using static typing with a scripting language like Python or Ruby. Dynamically typed code is 1/3rd the size of statically typed code, that means that one developer who is using dynamic typing is equivalent to 3 developers using statically typed code via MyPy. Since the code is 1/3rd of the size it contains 1/3rd of the bugs. This is confirmed by all the studies that have been done on the topic. If you use a static type checking with Python, you have increased your development time by 3 and your bug count by 3. Static typing's advantage is that the code runs a lot faster but that's only true if the language itself is statically typed. So with Python you have just screwed up.
- bombolo 4y ago
  > Dynamically typed code is 1/3rd the size of statically typed code, This is absolutely not true. > Since the code is 1/3rd of the size it contains 1/3rd of the bugs. That is made up and contrary to all empirical evidence I've ever collected. I'd be curious if you have a source, but I doubt it.
  - ReflectedImage 4y ago
    Anyone with experience of writing both dynamic typed and statically typed can tell you that. Infact, you could just try it out for yourself. But here is your internet source for this blatantly obvious fact: https://games.greggman.com/game/dynamic-typing-static-typing/ https://games.greggman.com/game/dynamic-typing-static-typing...

c120 4y ago

Not sure if I like the recommendation to not let Black change your code and just give out errors. I absolutely let Black change code and see the value in Black that it does that so the devs do not have to spend time on manually formatting code. Black shouldn't break anything (and hasn't broken anything for me in the years I used it) but in the unlikely case it does it, there's still pytests/unittests after that that should catch problems...

bsdz 4y ago
IIRC, Black also checks byte code before and after formatting to ensure source code functionality is unaffected.
- progval 4y ago
  Black checks the AST, not the byte code.
dustrider 4y ago
As I understood it, it was to not let black do the formatting during CI builds. In local dev you’d let it reformat. Even while it won’t break anything you want CI to be your safety net, flagging a local setup as being wrong is more valuable than magically autocorrecting it.
- nine_k 4y ago
  We have Black as a pre-commit hook; works fine, even if it disagrees with your IDE a little bit sometimes. CI/CD has no business changing your code; it builds stuff using it, exactly as if commit such-and-such.
  - 411111111111111 4y ago
    > CI/CD has no business changing your code; it builds stuff using it, exactly as if commit such-and-such. That going too far unless you define code to be a subset of the files checked into the repository and simply define any file that's touched in an automated manner to be not code There are a lot of useful automations that can be part of the CI/CD pipeline, such as increasing a version number, generating a changelog, creating new deployment configuration etc They don't have to be part of it and it's possible to work around it/don't commit... But that comes with it's own challenges and issues

aitchnyu 4y ago

I wish VSCode would figure out that ExampleModel.objects.first() returns ExampleModel or None or ExampleModel.objects.filter() returns an iterable of ExampleModel. Has anybody gotten this working, automatically or manually annotating?

jsmeaton 4y ago
You can annotate the manager and get some typing help in the editor. And there’s django-stubs which helps a little when running mypy. It’s not as good as pycharm though. https://github.com/typeddjango/django-stubs/tree/master https://github.com/typeddjango/django-stubs/tree/master
- aitchnyu 4y ago
  Could you share a guide on that?
  - jsmeaton 4y ago
    I don't have anything specific but here's something I quickly threw together demonstrating what I mean https://gist.github.com/jarshwah/1e683416d2ed2df28f254fc787d0d304 https://gist.github.com/jarshwah/1e683416d2ed2df28f254fc787d...
jerrygenser 4y ago
It's not a shortcoming of vscode it's due to the dynamic untyped nature of Django models unless you have a plugin or add typing to your own managers

mau 4y ago

One thing that is underestimated is keep the tools version in sync between your app dev dependencies and pre-commit. This also includes plugins for specific tools (for instance flake8). A solution would be to define the hooks in pre-commit to run the tools inside your venv. About typings: I agree the eco-system is not mature enough, especially for some frameworks such as Django, but the effort is still valuable and in many cases the static analysis provided by mypy is more useful than not using it at all. So I would suggest to try do your best to make it work.

modeopfer 4y ago

What's the current state of the art of managing multiple virtual environments, running tests and running your application? On Ubuntu and Windows I use Poetry [0], and it works, although it has (had?) some quirks during the installation on Windows. I liked its portability and lockfile format though. A few years ago I used conda [1], which was nice because it came batteries included especially for Deep Learning stuff. I switched because it felt way to heavy for porting scripts and small applications to constrained devices like a Raspberry Pi. And then there are also Docker Images, which I use if I want to give an application to somebody that "just works". What's your method of choice? [0] https://python-poetry.org/ https://python-poetry.org/ [1] https://www.anaconda.com/ https://www.anaconda.com/

wil421 4y ago
>I switched because it felt way to heavy for porting scripts and small applications to constrained devices like a Raspberry Pi. Agreed. I like docker images for smallish portable scripts. At home I can develop on my Mac and port it to a Raspberry PI or another x86 Windows/Linux box. Planning on running a docker swarm with a few Pi’s to see how it works.
gjulianm 4y ago
I use pip-tools to build a requirements.txt file from a requirements.in file. It does basically the same as poetry, but more manually. For me that's good because one of the application has a lot of requirements, and it needs to be deployed on systems with different Python versions, and the requirements need to be packaged along with the application because the servers have very limited internet access. So as long as Poetry doesn't add good support for multiple python versions and/or easy packaging of all dependencies, it isn't worth it for me to do the migration.
Flex247A 4y ago
If you feel that Anaconda is too heavy, try Miniconda [0]. The base environment is a standard Python 3.9 environment without any additional packages. [0] https://docs.conda.io/en/latest/miniconda.html https://docs.conda.io/en/latest/miniconda.html

jerrygenser 4y ago

Even since the start of python typing, it was recommended to use a more generic type like Iterable instead of List. The author claims that List is too specific -- this seems like a straw man argument against typing that doesn't acknowledge python's own advice. Also, mypy has gotten really good in recent years and I can vouch that on projects that have typing I catch bugs much much sooner. Previously I would only catch bugs when unit testing, now they are much more commonly type errors. The other thing typing does is allow for refactoring code. If anything, high code quality relates to the ability to refactor code confidently and typing helps this. Therefore I would put it at the top of the list above all the tooling presented (exception I agree with ci/cd)

whalesalad 4y ago
> Even since the start of python typing, it was recommended to use a more generic type like Iterable instead of List. The author claims that List is too specific These statements contradict themselves? List is too specific, and Sequence[item] is preferred. Sometimes you are dealing with a tuple, or a generator, and so it makes more sense to annotate that it is a generic iterable versus a concrete list.
- jerrygenser 4y ago
  From the original article: > For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it. Yet the Python type-annotation ecosystem was strongly oriented around nominal typing (i.e., caring that something is exactly a list) from the beginning. I'm saying that this quote is a straw man and that contrary to what is claimed in the quote, instead, the ecosystem would go with/recommend Iterable[Item] or Sequence[Item] and not List[Item] if applicable. I think we both agree, not sure which part of my comment you think is contradictory.
- hbrn 4y ago
  Whether something is generic/specific depends on the context. As an argument type, Iterable is permissive (generic). As a return type, Iterable is restrictive (specific).

IshKebab 4y ago

> For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it. Terrible advice not to use type hints and this reason makes no sense. There's already pretty good support for Sequence and Iterable and so on, and if you run into a place where you really can't write down the types (e.g. kwargs, which a lot of Python programmers abuse), then you can use Any. Blows my mind how allergic Python programmers are to static typing despite the huge and obvious benefits. It's true that Python's static typing does suck balls compared to most languages, but they're still a gazillion times better than nothing, and most of the reason they suck so much is that so many Python developers don't use them!