45 ms·

Great ideas. A lot of the freelance Python coding work I do is data transfer either from database to database, file to database, API to database, or some combin

Great ideas. A lot of the freelance Python coding work I do is data transfer either from database to database, file to database, API to database, or some combination thereof.

justaguyhere 8y ago
Can you give an example of file to database and API to database? As long as the file is a CSV, there are tons of tools already to import them into a DB, isn't it? Are you talking about unstructured files here?
- jakobegger 8y ago
  One thing I do is import data from Github issues into a table. There is no CSV export, but there is an API. So I first write a script that lists issues. They are paginated, so I need to keep making requests until I have all of them. Once I have a list of all the issues, I need to make one request per issue to get further details that aren't included in the original list. Then I need to extract data from all the downloaded JSON files and combine it into a table. It's conceptually not a hard problem, but it takes a lot of trial and error to do that if you aren't very familiar with Python. I wish there was a nice GUI where I could just click a few buttons to do that. Designing that GUI is probably not easy...
  - akudha 8y ago
    Woah, I wouldn't have guessed. I find it crazy that Github doesn't have a download all option, either in the UI or at least in the API. Any more examples you can share?
    - jakobegger 8y ago
      My online store has an API endpoint to get order details, but they don't have a way to list orders. So to get sales numbers for last week, I had to write a script that uses IMAP to log into my email account, checks the folder with order confirmation messages, parses the order numbers from the email subjects, then makes an API call for each order number to get order details. Then I get foreign exchange rates for every day of last week from a different API, put it all in a PostgreSQL database, and then I have a view that estimates weekly revenue...
      - justaguyhere 8y ago
        What platform does your store use? Thanks for replying, after reading your comments, I did a quick search. There are tools like Panoply, Blendo etc, but I couldn't find many with the level of customization that you described here.
        jakobegger 8y ago
        I was talking about Fastspring, they run my store, and they offer the API and send order confirmation emails. The IMAP thing is just what I use to get a list of orders. (They also support webhooks for incoming orders, but that would require me to run a webservice... the IMAP hack seemed easier) Edit: I looked at Blendo, it sounds like that's kindof what I'm looking for, but it's probably cloud based and they don't even have pricing on their website ... not exactly the lightweight tool I had in mind.
- zapperdapper 8y ago
  Generally it looks like: unstructured mess -> CSV -> database, or mess -> structured intermediate format -> whatever. Or database -> nicely formatted report that makes CEO look good. Even structured (weird XML) -> sort of structured (XHTML). There was no generic solution although I've ended up reusing a lot of code. One of the first gigs I got was to process about 7,000 pages from a static website, scrape out the relevant data, and plonk it in a database so they could run queries on it. :) You could probably latch onto a common use case and build a solution for that e.g. Blogger to Wordpress transfer - something I've had to do in the past by building some custom tools. (That was years ago though - I've not looked at current state of play with regards tool availability). Not suggesting that would be your use case though - just an example.

Great ideas. A lot of the freelance Python coding work I do is data transfer either from database to database, file to database, API to database, or some combination thereof.

justaguyhere 8y ago

Can you give an example of file to database and API to database? As long as the file is a CSV, there are tons of tools already to import them into a DB, isn't it? Are you talking about unstructured files here?

jakobegger 8y ago
One thing I do is import data from Github issues into a table. There is no CSV export, but there is an API. So I first write a script that lists issues. They are paginated, so I need to keep making requests until I have all of them. Once I have a list of all the issues, I need to make one request per issue to get further details that aren't included in the original list. Then I need to extract data from all the downloaded JSON files and combine it into a table. It's conceptually not a hard problem, but it takes a lot of trial and error to do that if you aren't very familiar with Python. I wish there was a nice GUI where I could just click a few buttons to do that. Designing that GUI is probably not easy...
- akudha 8y ago
  Woah, I wouldn't have guessed. I find it crazy that Github doesn't have a download all option, either in the UI or at least in the API. Any more examples you can share?
  - jakobegger 8y ago
    My online store has an API endpoint to get order details, but they don't have a way to list orders. So to get sales numbers for last week, I had to write a script that uses IMAP to log into my email account, checks the folder with order confirmation messages, parses the order numbers from the email subjects, then makes an API call for each order number to get order details. Then I get foreign exchange rates for every day of last week from a different API, put it all in a PostgreSQL database, and then I have a view that estimates weekly revenue...
    - justaguyhere 8y ago
      What platform does your store use? Thanks for replying, after reading your comments, I did a quick search. There are tools like Panoply, Blendo etc, but I couldn't find many with the level of customization that you described here.
      - jakobegger 8y ago
        I was talking about Fastspring, they run my store, and they offer the API and send order confirmation emails. The IMAP thing is just what I use to get a list of orders. (They also support webhooks for incoming orders, but that would require me to run a webservice... the IMAP hack seemed easier) Edit: I looked at Blendo, it sounds like that's kindof what I'm looking for, but it's probably cloud based and they don't even have pricing on their website ... not exactly the lightweight tool I had in mind.
zapperdapper 8y ago
Generally it looks like: unstructured mess -> CSV -> database, or mess -> structured intermediate format -> whatever. Or database -> nicely formatted report that makes CEO look good. Even structured (weird XML) -> sort of structured (XHTML). There was no generic solution although I've ended up reusing a lot of code. One of the first gigs I got was to process about 7,000 pages from a static website, scrape out the relevant data, and plonk it in a database so they could run queries on it. :) You could probably latch onto a common use case and build a solution for that e.g. Blogger to Wordpress transfer - something I've had to do in the past by building some custom tools. (That was years ago though - I've not looked at current state of play with regards tool availability). Not suggesting that would be your use case though - just an example.