7 ms·
Show HN: Auto-generate an OpenAPI spec by listening to localhost
Hey HN! We've developed OpenAPI AutoSpec, a tool for automatically generating OpenAPI specifications from localhost network traffic. It’s designed to simplify the creation of API documentation by just using your website or service, especially useful when you're pressed for time.
Documenting endpoints one by one sucks. This project originated from us needing it at our past jobs when building 3rd-party integrations.
It acts as a local server proxy that listens to your application’s HTTP traffic and automatically translates this into OpenAPI 3.0 specs, documenting endpoints, requests, and responses without much effort.
Installation is straightforward with NPM, and starting the server only requires a few command-line arguments to specify how and where you want your documentation generated ex. npx autospec --portTo PORT --portFrom PORT --filePath openapi.json
It's designed to work with any local website or application setup without extensive setup or interference with your existing code, making it flexible for different frameworks. We tried capturing network traffic on Chrome extension and it didn't help us catch the full picture of backend and frontend interactions.
We aim in future updates to introduce features like HTTPS and OpenAPI 3.1 specification support.
For more details and to get started, visit our GitHub page (https://github.com/Adawg4/openapi-autospec https://github.com/Adawg4/openapi-autospec). We also have a Discord community (https://discord.com/invite/CRnxg7uduH https://discord.com/invite/CRnxg7uduH) for support and discussions around using OpenAPI AutoSpec effectively.
We're excited to hear what you all think!
- 0cf8612b2e1e 2y agoCould this work with previously saved traffic dumps?
- leoqa 2y agoJust replay into nc?
- hnrodey 2y agoI thought I recalled something similar to this posted in the past. https://news.ycombinator.com/item?id=38012032 https://news.ycombinator.com/item?id=38012032 (5 months ago)
- adawg4 2y agoLove what Andrew is doing! We built this for localhost when testing vs. the client side with a Chrome extension.
- DidYaWipe 2y agoThis is probably cool and useful, but there's no way to know how much of the API you're covering, right? I find that the real shortage of tools exists going the other way: from OpenAPI to code. The ecosystem has proven to be a huge disappointment there, comprising a janky collection of defective tools that still (years later) don't support version 3.1 (a critical update).
- deleted 2y ago[deleted]
- ivan888 2y agoI think going from code to OpenAPI makes a lot more sense, at least for strong typed languages. And even if not directly translated from code, at least closer to the actual code, in annotations or something. Generating the spec from code removes a step, where you simply need to update code, rather than update the spec then update the code
- simplesager 2y agoCompletely agree. Keeping the openapi spec as tightly coupled to types in code builds a single source truth from your development to your deployment
- spxneo 2y agoI think pairing this tool with something that recursively clicks through app would be insanely helpful. (the latter is what I have trouble finding)
- adawg4 2y agoOpen to ideas! We're thinking of adding agents/crawler suggestions to the github if there's a package that clicks around in that fashion
- samstave 2y agoForgive the naive question, but to pair with the GP, thoughts: 1. Wouldnt this also be helpful in understanding the exact nature of all traffic/calls against a particular page, user-workflow matriculating through your site from a UX perspective? 2. Could one make a proxy from this on a local home egress such that you could see the nature of outbound network traffic to site you visit (more importantly, traffic heading to 3rd-party trackers/cookies' APIs via your site visits? 3. Could it be used to nefariously map open API endpoints against a system one is (whiteHat) pen testing?
- cipherself 2y agoOne slightly related thing you can do is to test the API with schemathesis[0] [0] https://github.com/schemathesis/schemathesis https://github.com/schemathesis/schemathesis
- knicholes 2y agoThis seems pretty simple to me to do. Search the html of the main page for anchor tags. Add the links in those tags to an array as your exploration frontier. Once done parsing that html, load the next link. Add deduplication to avoid loops and just run a depth-first search. What am I missing?
- dns_snek 2y agoIn many web apps there are going to be buttons and links that are not represented as <a>. You would realistically have to enumerate everything that has any kind of event handler attached since it could potentially trigger an API call. You would also have to fill and submit forms with valid and invalid data. You would have to toggle checkboxes, change radio buttons, click buttons, (e.g. "Apply filters" after changing values in a product filter section), and generally go through many combinations of inputs to find all valid parameters and possible responses.
- nattaylor 2y agoReminds me of https://github.com/alufers/mitmproxy2swagger https://github.com/alufers/mitmproxy2swagger which I discovered from this thread https://news.ycombinator.com/item?id=31354130 https://news.ycombinator.com/item?id=31354130 I generated some specs from that! I ran into trouble keeping them up to date.
- soneil 2y agoThis looks interesting - it seems like it wouldn't be a huge lift to turn this into something that compares against an existing spec too. create an AutoSpec, take your defined spec, and spot the difference.
- remoquete 2y agoA nice tool for research, or for documenting third-party APIs. Let's not forget, though, that one of the goals of OpenAPI is to serve as a design and documentation artifact in design-first API development; generating OpenAPI from code or, as in this case, from network traffic, is an interesting complement and something you can use to test the implementation against the design.
- adawg4 2y agoIt makes sense, and we love API-first companies. How are frameworks prioritizing this? Seen DRF and lite star but seemed like this was needed to help at places we worked/API market reports about companies that hadn't put those standards in yet
- arscan 2y agoI have a similar need but for the FHIR[1] spec, which has its own way of describing RESTful http endpoints that serve FHIR data. I was looking into how this works for inspiration, and it seems like the work of inferring the OpenAPI definition from recorded requests/responses is handled by the har-to-openapi nodejs library [2]. Is this by the same team? If not, kudos for packaging this up in a proxy -- seems like a useful interface into that library. 1. https://www.hl7.org/fhir/ https://www.hl7.org/fhir/ 2. https://github.com/jonluca/har-to-openapi https://github.com/jonluca/har-to-openapi
- deleted 2y ago[deleted]
- w3news 2y agoWhen you build an API, please start with the OpenAPI specification, before you write any code for your API. It can be iterative, but for every part, just start with the OpenAPI, and think about what you want from the API, what do you want to send, and what to receive. It is like the TDD approach, design before build. Writing or generating tests after you build the code, is the same as this. It is guessing what it should do. The OpenAPI specification, and the tests should tell you what it should do, not the code. If you have the specification, everyone (and also AI) can write the code for you to make it work. But the specification is about what you think it should do. That are the questions and requirements that you have about the system.
- physicsguy 2y agoI prefer going the other direction in practice, autogenerating the spec from the code e.g. with drf-spectacular for Django.
- paholg 2y agoI don't want to write OpenApi. Yaml is a terrible programming language, and keeping it in sync with actual code is always a nightmare. I've been using a tool to generate OpenApi from code, and am pretty happy with that workflow. Even if writing the API before logic, I'd much rather write the types and endpoints in a real programming language, and just have a `todo` in the body. You can still write API-driven code without literally writing OpenApi first.
- rnts08 2y agoAbsolutely, and yes YAML is trash.
- adawg4 2y agoWhich tool?
- bigiain 2y agoI'm thinking using this tool, and having your test suite run through it might work? At least for people comfortable with doing test driven development. Write your requirements for your API-driven code as tests first, then document those APIs by running the tests through this tool.
- deleted 2y ago[deleted]
- sbeckeriv 2y agoI wrote a tool for work that does the same thing based on request logs. It would parse each line into a structure then merge the same call point structures down to one spec. It was helpful to see the api but in the end was not that helpful in back filling the openapi spec. things to consider: - junk data from users will show up. unless your downstream service rejects extra params users will mess with you. - it documents each endpoint but its harder to say if this "user" data is the same as another's endpoints "user" - it is hard to tell if users are hitting all endpoint inputs/outputs without manual review.
- bigiain 2y agoDoesn't look like this tool is intended to be deployed into production where "junk data from users" would be encountered. My impression is it's a localhost proxy which only ever sees deliberate test traffic from the developer who's running it on their own machine. (Although I'd be curious to see something very similar to this running in prod and generating WAF rules and/or alerting on suspicious requests. Kinda like Dynatrace or Splunk, but much more aware of the API documentation and expectations.)
- rattray 2y agoAnother good option for this kind of thing is Optic: https://www.useoptic.com/docs/generate-openapi-from-tests https://www.useoptic.com/docs/generate-openapi-from-tests
- jzig 2y agoFeature request: a browser extension that intercepts HTTP requests for any given website (instead of localhost) and documents their API :)
- dxbednarczyk 2y agoSee https://github.com/AndrewWalsh/openapi-devtools https://github.com/AndrewWalsh/openapi-devtools
- jzig 2y agoVery cool. Thank you!
- raybb 2y agoThis is fantastic! I think I'll try to use this to generate the spec for openlibrary.org APIs. We have a few basic ones now but it's a huge pain to write. Someone looked into generating them from the python code but it didn't pan our.
- adawg4 2y agoThat would be awesome!
- radicality 2y agoI’ve only glanced at the code on mobile, but am I reading this right? It seems like this does pretty much… nothing apart from writing everything to a .har file, and then calls out to a separate library called “har-to-openapi” to do the actual work?https://github.com/jonluca/har-to-openapi https://github.com/jonluca/har-to-openapi
- rrr_oh_man 2y ago1. What else should it do? 2. I think I like this blunt elevator pitch much better than OP's multiple paragraphs of text...
- beardedwalleye 2y agoI could see this (or similar) being useful for generating a spec for an old, undocumented, legacy "service"
- tczMUFlmoNk 2y agoIs this comparable to Akita? https://www.akitasoftware.com/ https://www.akitasoftware.com/ > By watching API traffic live and automatically inferring endpoint structure, Akita bypasses the complexity of implementing and maintaining monitoring and observability. > […] > - Export your API description as an OpenAPI spec. (Not affiliated, nor am I a user of either of these.)
- dingi 2y agoAlthough this feels like a ok-ish approach to just serve API docs to users, you are missing the whole point of specs if you use this approach. I have started to extensively use spec-first approach with open API. You wouldn't believe how many hours it saved for me despite the initial cost of time to get things started. For example, No need to tediously write DTOs ever, just have them generated using API spec. Need an SDK for API consumers, just generate an SDK using the spec for almost all the popular languages. While using Open API as just an API documentation is fine, it is a waste of potential that Open API provides.