5 ms·

New tool from curl creator – trurl – for URL parsing and manipulation

140 points by michalg82 3y ago

Groxx 3y ago
Might be more useful to link to the blog post: https://daniel.haxx.se/blog/2023/04/03/introducing-trurl/ https://daniel.haxx.se/blog/2023/04/03/introducing-trurl/
- kencausey 3y ago
  Previously posted: https://news.ycombinator.com/item?id=35419573 https://news.ycombinator.com/item?id=35419573 - 22 points, 2 comments https://news.ycombinator.com/item?id=35426050 https://news.ycombinator.com/item?id=35426050 - 5 points, 0 comments
- dang 3y ago
  Ok, we've changed the URL to that from https://github.com/curl/trurl https://github.com/curl/trurl. Thanks!
neuracnu 3y ago
The readme should include a pronunciation guide for the name. tru-rul? tee-ru-rul?
- jayrhynas 3y ago
  > We say "trurel". As if there was an 'e' between the r and l at the end. From https://curl.se/trurl/ https://curl.se/trurl/
- cadizm 3y ago
  Maybe "tee" "arr" url? I'm guessing the "tr" is for https://en.wikipedia.org/wiki/Tr_(Unix) https://en.wikipedia.org/wiki/Tr_(Unix)
- b4ke 3y ago
  where did the extra syllable appear from? like tr -ee and -c url trurl.
- pimlottc 3y ago
  I would say, "rhymes with rural"
- marban 3y ago
  Trurl is the name of a character in the science fiction short story collection "The Cyberiad" by Polish author Stanisław Lem. Trurl is a highly intelligent robot and inventor who, along with his friend Klapaucius, goes on various adventures throughout the universe. — via CGPT
  - myhf 3y ago
    A great book that prefigured the kinds of interactions people are now having with ChatGPT. For example: “Have it compose a poem — a poem about a haircut! But lofty, noble, tragic, timeless, full of love, treachery, retribution, quiet heroism in the face of certain doom! Six lines, cleverly rhymed, and every word beginning with the letter S!!” “Seduced, shaggy Samson snored. She scissored short. Sorely shorn, Soon shackled slave, Samson sighed, Silently scheming, Sightlessly seeking Some savage, spectacular suicide.”
    - omoikane 3y ago
      Stanisław Lem’s "Cyberiad" is a great book that predicted machine learning and various modern technologies, although credit for this particular poem should probably go to Michael Kandel for his brilliant English translation. It's very different in other languages: https://mwichary.medium.com/seduced-shaggy-samson-snored-725b5a8086d9 https://mwichary.medium.com/seduced-shaggy-samson-snored-725...
    - F2hP18Foam 3y ago
      I had a feeling that ChatGPT would have trouble with some aspects of a request that specific, but here's what it gave me: She sat with scissors, ready to snip, Slicing strands of hair, curl by curl, clip by clip, Shears shimmering, sharp as swords in sunlight, Swiftly, skillfully, she shortened the sight, Saddened by strands strewn on the floor, She stood, serene, satisfied, the style was no more. Clearly nowhere as good, and it couldn't get all the words to start with an 'S', even with a few attempts to correct it. Still, some software should succeed someday, sans skipped S's. Substituting some synonyms, surely.
- otikik 3y ago
  You pronnounce it like curl, but changing how you pronounce "c" by how you pronounce "tr".
gadrev 3y ago
https://github.com/curl/trurl https://github.com/curl/trurl Apparently, it requires curl 7.81 to compile, but I'm on ubuntu 20 (7.6x) and wanted to do a quick try, so this patch makes it work (trivially) by removing the couple of new symbols depending on that newer version. Just a quick hack to compile. --- a/trurl.c +++ b/trurl.c @@ -362,11 +362,12 @@ static void get(struct option *op, CURLU *uh) case CURLUE_NO_PORT: case CURLUE_NO_QUERY: case CURLUE_NO_FRAGMENT: - case CURLUE_NO_ZONEID: + // would require 7.81 + // case CURLUE_NO_ZONEID: /\* silently ignore */ break; default: - fprintf(stderr, PROGNAME ": %s (%s)\n", curl_url_strerror(rc), + fprintf(stderr, PROGNAME ": %s (%s)\n", rc, variables[i].name); break; } @@ -589,8 +590,8 @@ static void singleurl(struct option *o, CURLU_GUESS_SCHEME|CURLU_NON_SUPPORT_SCHEME); if(rc) { if(o->verify) - errorf(ERROR_BADURL, "%s [%s]", curl_url_strerror(rc), url); - warnf("%s [%s]", curl_url_strerror(rc), url); + errorf(ERROR_BADURL, "%s [%s]", rc, url); + warnf("%s [%s]", rc, url); } else { if(o->redirect)
obelos 3y ago
Oh, c'mon, “purl” was right there!
- sidpatil 3y ago
  That would collide with this: https://en.m.wikipedia.org/wiki/Persistent_uniform_resource_locator https://en.m.wikipedia.org/wiki/Persistent_uniform_resource_...
- Izkata 3y ago
  Given what it does, and to avoid confusion with perl, "transform url" seems more reasonable. Though I probably would have gone with "turl" so it's easier to pronounce.
  - enneff 3y ago
    It seems like a nod to the “tr” command.
    - jcul 3y ago
      Yup, I guessed the same, and indeed it explicitly states so in the blog post linked by another commenter. > trurl is a tool in a similar spirit of tr but for URLs. Here, tr stands for translate or transpose.
    - obelos 3y ago
      Though it hardly gets any use because s/// is so much more flexible, Perl also has a tr/// builtin that replicates the behavior of the command line tool.
- twic 3y ago
  I wonder if the name is a Lem reference: https://en.wikipedia.org/wiki/The_Cyberiad https://en.wikipedia.org/wiki/The_Cyberiad
kjellsbells 3y ago
Pretty dang cool, but I might have missed a feature: can trurl do multiple manipulations on a single url, as opposed to a single manipulation on multiple urls (which the blog post says is supported)? For example, you want to apply a sequence of predefined normalization steps, such as removing the user part, converting http to https, etc etc. You put these steps in a file and then invoke trurl with that file and pointed at your url or urls. Very much like what you would do in sed, say. Possible?
- skibz 3y ago
  I think it can do what you're suggesting, if I'm understanding correctly: > $ trurl --url https://example.com https://example.com --set path=/abc/123 --set scheme=ftp --set host=foo.bar > $ ftp://foo.bar/abc/123
theamk 3y ago
that an ... interesting ... decsion to put inputs before -- and commands/options after -- Most of the tools assume opposite default (like xargs for example). Sure, you can specify a different order but why make life harder for no reason?
- ftrobro 3y ago
  Usage: " PROGNAME " [options] [URL] But you can put options wherever you want, they are identified by the leading '-' char.
Goofy_Coyote 3y ago
Pardone my ignorance, serious question: Why is this a big deal? I'm probably underestimting the work needed, but it doesn't look like a hard thing to write. What am I missing?
- lultimouomo 3y ago
  From the blog post introducing trurl: > URLs are tricky to parse and there are numerous security problems in software because of this. trurl wants to help soften this problem by taking away the need for script and command line authors everywhere to re-invent the wheel over and over.
- Etheryte 3y ago
  Over the years, curl itself has had 9 CVEs relating to handling URLs [0] so this is most definitely not a trivial piece of code to write. The basic case is easy, yes. Getting everything in the spec right and then some is hard. [0] https://curl.se/docs/security.html https://curl.se/docs/security.html
  - skrtskrt 3y ago
    is it somehow better than the stuff available in mainstream languages (go net/url, Python yarl, etc)? Or is it just that this is focused on command-line usage
  - organsnyder 3y ago
    I had to implement a routine involving URL parsing in a library that is supposed to be behavior-identical across implementations in multiple languages. That was fun.
  - gcoakes 3y ago
    And, now I'm scared of establishing a TLS connection with untrusted servers after reading CVE-2021-22901 from that page. Remote code execution from an adversarial *server*. I can understand an adversarial client, but that just expanded the things of which I'm wary.
- specproc 3y ago
deleted 3y ago
[deleted]
mkraft 3y ago
Only two unit tests?
- mkraft 3y ago
  I’m curious about why people dislike this question. This is a specialized library; it should be testing everything including crazy edge cases that most of us haven’t even considered. Also, someone stated that there have been several URL-related security vulnerabilities in curl.

140 points by michalg82 3y ago

Might be more useful to link to the blog post: https://daniel.haxx.se/blog/2023/04/03/introducing-trurl/ https://daniel.haxx.se/blog/2023/04/03/introducing-trurl/

kencausey 3y ago
Previously posted: https://news.ycombinator.com/item?id=35419573 https://news.ycombinator.com/item?id=35419573 - 22 points, 2 comments https://news.ycombinator.com/item?id=35426050 https://news.ycombinator.com/item?id=35426050 - 5 points, 0 comments
dang 3y ago
Ok, we've changed the URL to that from https://github.com/curl/trurl https://github.com/curl/trurl. Thanks!

neuracnu 3y ago

The readme should include a pronunciation guide for the name. tru-rul? tee-ru-rul?

jayrhynas 3y ago
> We say "trurel". As if there was an 'e' between the r and l at the end. From https://curl.se/trurl/ https://curl.se/trurl/
cadizm 3y ago
Maybe "tee" "arr" url? I'm guessing the "tr" is for https://en.wikipedia.org/wiki/Tr_(Unix) https://en.wikipedia.org/wiki/Tr_(Unix)
b4ke 3y ago
where did the extra syllable appear from? like tr -ee and -c url trurl.
pimlottc 3y ago
I would say, "rhymes with rural"
marban 3y ago
Trurl is the name of a character in the science fiction short story collection "The Cyberiad" by Polish author Stanisław Lem. Trurl is a highly intelligent robot and inventor who, along with his friend Klapaucius, goes on various adventures throughout the universe. — via CGPT
- myhf 3y ago
  A great book that prefigured the kinds of interactions people are now having with ChatGPT. For example: “Have it compose a poem — a poem about a haircut! But lofty, noble, tragic, timeless, full of love, treachery, retribution, quiet heroism in the face of certain doom! Six lines, cleverly rhymed, and every word beginning with the letter S!!” “Seduced, shaggy Samson snored. She scissored short. Sorely shorn, Soon shackled slave, Samson sighed, Silently scheming, Sightlessly seeking Some savage, spectacular suicide.”
  - omoikane 3y ago
    Stanisław Lem’s "Cyberiad" is a great book that predicted machine learning and various modern technologies, although credit for this particular poem should probably go to Michael Kandel for his brilliant English translation. It's very different in other languages: https://mwichary.medium.com/seduced-shaggy-samson-snored-725b5a8086d9 https://mwichary.medium.com/seduced-shaggy-samson-snored-725...
  - F2hP18Foam 3y ago
    I had a feeling that ChatGPT would have trouble with some aspects of a request that specific, but here's what it gave me: She sat with scissors, ready to snip, Slicing strands of hair, curl by curl, clip by clip, Shears shimmering, sharp as swords in sunlight, Swiftly, skillfully, she shortened the sight, Saddened by strands strewn on the floor, She stood, serene, satisfied, the style was no more. Clearly nowhere as good, and it couldn't get all the words to start with an 'S', even with a few attempts to correct it. Still, some software should succeed someday, sans skipped S's. Substituting some synonyms, surely.
otikik 3y ago
You pronnounce it like curl, but changing how you pronounce "c" by how you pronounce "tr".

gadrev 3y ago

https://github.com/curl/trurl https://github.com/curl/trurl Apparently, it requires curl 7.81 to compile, but I'm on ubuntu 20 (7.6x) and wanted to do a quick try, so this patch makes it work (trivially) by removing the couple of new symbols depending on that newer version. Just a quick hack to compile. --- a/trurl.c +++ b/trurl.c @@ -362,11 +362,12 @@ static void get(struct option *op, CURLU *uh) case CURLUE_NO_PORT: case CURLUE_NO_QUERY: case CURLUE_NO_FRAGMENT: - case CURLUE_NO_ZONEID: + // would require 7.81 + // case CURLUE_NO_ZONEID: /\* silently ignore */ break; default: - fprintf(stderr, PROGNAME ": %s (%s)\n", curl_url_strerror(rc), + fprintf(stderr, PROGNAME ": %s (%s)\n", rc, variables[i].name); break; } @@ -589,8 +590,8 @@ static void singleurl(struct option *o, CURLU_GUESS_SCHEME|CURLU_NON_SUPPORT_SCHEME); if(rc) { if(o->verify) - errorf(ERROR_BADURL, "%s [%s]", curl_url_strerror(rc), url); - warnf("%s [%s]", curl_url_strerror(rc), url); + errorf(ERROR_BADURL, "%s [%s]", rc, url); + warnf("%s [%s]", rc, url); } else { if(o->redirect)

obelos 3y ago

Oh, c'mon, “purl” was right there!

sidpatil 3y ago
That would collide with this: https://en.m.wikipedia.org/wiki/Persistent_uniform_resource_locator https://en.m.wikipedia.org/wiki/Persistent_uniform_resource_...
Izkata 3y ago
Given what it does, and to avoid confusion with perl, "transform url" seems more reasonable. Though I probably would have gone with "turl" so it's easier to pronounce.
- enneff 3y ago
  It seems like a nod to the “tr” command.
  - jcul 3y ago
    Yup, I guessed the same, and indeed it explicitly states so in the blog post linked by another commenter. > trurl is a tool in a similar spirit of tr but for URLs. Here, tr stands for translate or transpose.
  - obelos 3y ago
    Though it hardly gets any use because s/// is so much more flexible, Perl also has a tr/// builtin that replicates the behavior of the command line tool.
twic 3y ago
I wonder if the name is a Lem reference: https://en.wikipedia.org/wiki/The_Cyberiad https://en.wikipedia.org/wiki/The_Cyberiad

kjellsbells 3y ago

Pretty dang cool, but I might have missed a feature: can trurl do multiple manipulations on a single url, as opposed to a single manipulation on multiple urls (which the blog post says is supported)? For example, you want to apply a sequence of predefined normalization steps, such as removing the user part, converting http to https, etc etc. You put these steps in a file and then invoke trurl with that file and pointed at your url or urls. Very much like what you would do in sed, say. Possible?

skibz 3y ago
I think it can do what you're suggesting, if I'm understanding correctly: > $ trurl --url https://example.com https://example.com --set path=/abc/123 --set scheme=ftp --set host=foo.bar > $ ftp://foo.bar/abc/123

theamk 3y ago

that an ... interesting ... decsion to put inputs before -- and commands/options after -- Most of the tools assume opposite default (like xargs for example). Sure, you can specify a different order but why make life harder for no reason?

ftrobro 3y ago
Usage: " PROGNAME " [options] [URL] But you can put options wherever you want, they are identified by the leading '-' char.

Goofy_Coyote 3y ago

Pardone my ignorance, serious question: Why is this a big deal? I'm probably underestimting the work needed, but it doesn't look like a hard thing to write. What am I missing?

lultimouomo 3y ago
From the blog post introducing trurl: > URLs are tricky to parse and there are numerous security problems in software because of this. trurl wants to help soften this problem by taking away the need for script and command line authors everywhere to re-invent the wheel over and over.
Etheryte 3y ago
Over the years, curl itself has had 9 CVEs relating to handling URLs [0] so this is most definitely not a trivial piece of code to write. The basic case is easy, yes. Getting everything in the spec right and then some is hard. [0] https://curl.se/docs/security.html https://curl.se/docs/security.html
- skrtskrt 3y ago
  is it somehow better than the stuff available in mainstream languages (go net/url, Python yarl, etc)? Or is it just that this is focused on command-line usage
- organsnyder 3y ago
  I had to implement a routine involving URL parsing in a library that is supposed to be behavior-identical across implementations in multiple languages. That was fun.
- gcoakes 3y ago
  And, now I'm scared of establishing a TLS connection with untrusted servers after reading CVE-2021-22901 from that page. Remote code execution from an adversarial *server*. I can understand an adversarial client, but that just expanded the things of which I'm wary.
specproc 3y ago

deleted 3y ago

[deleted]

mkraft 3y ago

Only two unit tests?

mkraft 3y ago
I’m curious about why people dislike this question. This is a specialized library; it should be testing everything including crazy edge cases that most of us haven’t even considered. Also, someone stated that there have been several URL-related security vulnerabilities in curl.