I have an gawk program that takes as input Events Stream

$CURL -s https://domain.com/path-to-stream | prog.awk

prog.awk
----
while ((getline line < "/dev/stdin") > 0) {
<does a lot of stuff>
}

So far it is working great (yay awk), but I wonder because "does a lot of stuff" might take a while some seconds, and I don't want to drop any incoming stream, which can be a lot. Not even sure what is doing the buffering - the OS via the pipe? Curl? Awk? Can data be lost without knowing? Thanks for any insight.

On 12.06.2021 07:33, Stephen wrote:
> I have an gawk program that takes as input Events Stream
>
> $CURL -s https://domain.com/path-to-stream | prog.awk
>
> prog.awk
> ----
> while ((getline line < "/dev/stdin") > 0) {
> <does a lot of stuff>
> }

This fragment (with <does a lot of stuff> expanded) is not a correct
awk program. (Is that code in a BEGIN block? Is there a #! line at
top?)

Why don't you just write

{ line = $0 ; <does a lot of stuff> }

> So far it is working great (yay awk), but I wonder because "does a
> lot of stuff" might take a while some seconds, and I don't want to
> drop any incoming stream, which can be a lot. Not even sure what is
> doing the buffering - the OS via the pipe? Curl? Awk? Can data be
> lost without knowing? Thanks for any insight.
>

What format has the "Events Stream" data? Is it line-oriented text?
Binary data? (Is the awk text processor the right tool for that data?)

All data from curl (assuming that is what's hidden in the CURL shell
variable) will be passed to awk, neither the OS nor awk "drop" data
passed through the pipe.

Janis

On 6/12/2021 12:33 AM, Stephen wrote:
> I have an gawk program that takes as input Events Stream
>
> $CURL -s https://domain.com/path-to-stream | prog.awk
>
> prog.awk
> ----
> while ((getline line < "/dev/stdin") > 0) {
> <does a lot of stuff>
> }
>
> So far it is working great (yay awk), but I wonder because "does a lot of stuff" might take a while some seconds, and I don't want to drop any incoming stream, which can be a lot. Not even sure what is doing the buffering - the OS via the pipe? Curl? Awk? Can data be lost without knowing? Thanks for any insight.
>

If "does a lot of stuff" takes some seconds then that "stuff" is almost
certainly not just manipulating text and so I suspect you're trying to
use awk like a shell and calling external tools from it. That's probably
a bad idea. If you'd like help post a minimal, complete script along
with concise, testable sample input (output of curl) and expected output
so we can help you.

Ed.