------=_Part_1568_998848939.1542824437272
Content-Type: multipart/alternative;
 boundary="----=_Part_1569_2000422299.1542824437272"

------=_Part_1569_2000422299.1542824437272
Content-Type: text/plain; charset="UTF-8"

Hi all,

I have this idea about a set of primitives for processing elements coming
from never-ending streams in C++. The main class is stream<T> with the
following methods:
  - stream<U> map(function<U(T&)> mapper): map a stream<T> to stream<U>
  - void foreach(function<void(T&)> processor): run the given function on
each element of the source stream.
  - void write_to(sink<T>& dest): write all elements to the given sink.
  - and so on.

All the elements are processed asynchronously, in parallel, using either an
Executor (coming feature of C++) or std::async.
The usage will be something like this:

stream<MyData> source(...);
sink<TransformedData> destination(...);
source.map(...).filter(...).map(...).write_to(destination);

I found these constructs very helpful for my usecase, where I process
messages coming from a distributed queue - something similar to Apache
Kafka.
The logic of processing messages can now be expressed using a chain of
transformations. I don't need to worry about maintaining a threadpool,
fibers, and all that threading operations.

I think it is going to be useful for other usecases as well, especially
when we allow users to create their own sources.
Is there anything similar like that in the standard? Do you think something
like that will be helpful?

Relately, I also have another design for parallelizing elements from
standard collections (vector, list, map, ...).
Unlike streams, those elements are countable and stored in-memory, so we
can define many other aggreation operations like reduce, aggregate...
Using these primitives, user can easily parallelize algorithms using
functional constructs, without maintaining low-level threading constructs.

I believe this is a useful low-overhead abstraction that will make people
love C++ even more!

I would love to discuss the details.

Cheers,
Vu

P.S. This is my first email to this group. I am following guides from your
website. Please forward it to the right channel if this is not the correct
place :)

--
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/067d8f3c-a700-4a4b-9ca7-6596480b20e9%40isocpp.org.

------=_Part_1569_2000422299.1542824437272
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi all,</div><div><br></div><div></div><div>I have th=
is idea about a set of primitives for processing elements coming from never=
-ending streams in C++. The main class is <span style=3D"font-family: couri=
er new, monospace;">stream&lt;T&gt;</span> with the following methods:</div=
><div>=C2=A0 - <span style=3D"font-family: courier new, monospace;">stream&=
lt;U&gt; map(function&lt;U(T&amp;)&gt; mapper)</span>: map a <span style=3D=
"font-family: courier new, monospace;">stream&lt;T&gt;</span> to <span styl=
e=3D"font-family: courier new, monospace;">stream&lt;U&gt;</span></div><div=
>=C2=A0 - <span style=3D"font-family: courier new, monospace;">void foreach=
(function&lt;void(T&amp;)&gt; processor)</span>: run the given function on =
each element of the source stream.</div><div>=C2=A0 - <span style=3D"font-f=
amily: courier new, monospace;">void write_to(sink&lt;T&gt;&amp; dest)</spa=
n>: write all elements to the given sink.</div><div>=C2=A0 - and so on.</di=
v><div><br></div><div>All the elements are processed asynchronously, in par=
allel, using either an Executor (coming feature of C++) or <span style=3D"f=
ont-family: courier new, monospace;">std::async</span>.</div><div>The usage=
 will be something like this:</div><div><br></div><div><span style=3D"font-=
family: courier new, monospace;">stream&lt;MyData&gt; source(...);</span></=
div><div><span style=3D"font-family: courier new, monospace;">sink&lt;Trans=
formedData&gt; destination(...);</span></div><div><span style=3D"font-famil=
y: courier new, monospace;">source.map(...).filter(...).map(...).write_to(d=
estination);</span></div><div><br></div><div>I found these constructs very =
helpful for my usecase, where I process messages coming from a distributed =
queue - something similar to Apache Kafka.</div><div>The logic of processin=
g messages can now be expressed using a chain of transformations. I don&#39=
;t need to worry about maintaining a threadpool, fibers, and all that threa=
ding operations.<br></div><div><br></div><div>I think it is going to be use=
ful for other usecases as well, especially when we allow users to create th=
eir own sources.</div><div>Is there anything similar like that in the stand=
ard? Do you think something like that will be helpful?</div><div><br></div>=
<div>Relately, I also have another design for parallelizing elements from s=
tandard collections (vector, list, map, ...). <br></div><div>Unlike streams=
, those elements are countable and stored in-memory, so we can define many =
other aggreation operations like <span style=3D"font-family: courier new, m=
onospace;">reduce</span>, <span style=3D"font-family: courier new, monospac=
e;">aggregate</span>...</div><div>Using these primitives, user can easily p=
arallelize algorithms using functional constructs, without maintaining low-=
level threading constructs.</div><div><br></div><div>I believe this is a us=
eful low-overhead abstraction that will make people love C++ even more!<br>=
</div><div><br></div><div>I would love to discuss the details.</div><div><b=
r></div><div>Cheers,</div><div>Vu</div><div><br></div><div>P.S. This is my =
first email to this group. I am following guides from your website. Please =
forward it to the right channel if this is not the correct place :)<br></di=
v><div><br></div></div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org">std-proposa=
ls+unsubscribe@isocpp.org</a>.<br />
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org">std-proposals@isocpp.org</a>.<br />
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/067d8f3c-a700-4a4b-9ca7-6596480b20e9%=
40isocpp.org?utm_medium=3Demail&utm_source=3Dfooter">https://groups.google.=
com/a/isocpp.org/d/msgid/std-proposals/067d8f3c-a700-4a4b-9ca7-6596480b20e9=
%40isocpp.org</a>.<br />

------=_Part_1569_2000422299.1542824437272--

------=_Part_1568_998848939.1542824437272--

.

--00000000000069bbe6057b358482
Content-Type: text/plain; charset="UTF-8"

> Do you think something like that will be helpful?

Definitely. Your idea is very powerful.
However I'm not sure it is something that should be part of the standard
library.
The reason is: There are many different ways to implement functional
reactive(FRP) data processing pipelines, each with very different
properties.

One of the most popular FRP instances are the 'reactive extensions'
libraries, which are implemented in almost every language:
RxCpp <https://github.com/ReactiveX/RxCpp> is the C++ version and a nice
visualisation of it's operators can be found here <http://rxmarbles.com/>.
RxCpp also has experimental support for the Coroutine TS. Checkout
<https://github.com/ReactiveX/RxCpp/blob/master/Rx/v2/examples/awaitable/main.cpp>
their example.

[image: Mailtrack]
<https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&>
Sender
notified by
Mailtrack
<https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&>
11/21/18,
11:56:45 PM

On Wed, 21 Nov 2018 at 18:20, Vu Pham <phvu225@gmail.com> wrote:

> Hi all,
>
> I have this idea about a set of primitives for processing elements coming
> from never-ending streams in C++. The main class is stream<T> with the
> following methods:
>   - stream<U> map(function<U(T&)> mapper): map a stream<T> to stream<U>
>   - void foreach(function<void(T&)> processor): run the given function on
> each element of the source stream.
>   - void write_to(sink<T>& dest): write all elements to the given sink.
>   - and so on.
>
> All the elements are processed asynchronously, in parallel, using either
> an Executor (coming feature of C++) or std::async.
> The usage will be something like this:
>
> stream<MyData> source(...);
> sink<TransformedData> destination(...);
> source.map(...).filter(...).map(...).write_to(destination);
>
> I found these constructs very helpful for my usecase, where I process
> messages coming from a distributed queue - something similar to Apache
> Kafka.
> The logic of processing messages can now be expressed using a chain of
> transformations. I don't need to worry about maintaining a threadpool,
> fibers, and all that threading operations.
>
> I think it is going to be useful for other usecases as well, especially
> when we allow users to create their own sources.
> Is there anything similar like that in the standard? Do you think
> something like that will be helpful?
>
> Relately, I also have another design for parallelizing elements from
> standard collections (vector, list, map, ...).
> Unlike streams, those elements are countable and stored in-memory, so we
> can define many other aggreation operations like reduce, aggregate...
> Using these primitives, user can easily parallelize algorithms using
> functional constructs, without maintaining low-level threading constructs.
>
> I believe this is a useful low-overhead abstraction that will make people
> love C++ even more!
>
> I would love to discuss the details.
>
> Cheers,
> Vu
>
> P.S. This is my first email to this group. I am following guides from your
> website. Please forward it to the right channel if this is not the correct
> place :)
>
> --
> You received this message because you are subscribed to the Google Groups
> "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to std-proposals+unsubscribe@isocpp.org.
> To post to this group, send email to std-proposals@isocpp.org.
> To view this discussion on the web visit
> https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/067d8f3c-a700-4a4b-9ca7-6596480b20e9%40isocpp.org
> <https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/067d8f3c-a700-4a4b-9ca7-6596480b20e9%40isocpp.org?utm_medium=email&utm_source=footer>
> .
>


--
Regards,

Gaetano Checinski
Founder of Loopperfect
https://loopperfect.com
https://buckaroo.pm

--
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAN0NuG5OsBskD-X2m3x%3DpTKJUEd9-WiMxV3MKJWRnK9YRnpZGA%40mail.gmail.com.

--00000000000069bbe6057b358482
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><img width=3D"0" height=3D"0" class=3D"mailtrack-img" alt=
=3D"" style=3D"display:flex" src=3D"https://mailtrack.io/trace/mail/220c2f9=
1f52dda91747683df1854ebfbc68d3820.png?u=3D931501"><div></div><div><br></div=
><div>&gt; Do you think something like that will be helpful?<br></div><div>=
<br></div><div>Definitely. Your idea is very powerful.=C2=A0</div><div>Howe=
ver I&#39;m not sure it is something that should be part of the standard li=
brary.</div><div>The reason is: There are many different ways to implement =
functional reactive(FRP) data processing pipelines, each with very differen=
t properties.</div><div><br></div><div>One of the most popular FRP instance=
s are the &#39;reactive extensions&#39; libraries, which are implemented in=
 almost every language:</div><div><a href=3D"https://github.com/ReactiveX/R=
xCpp">RxCpp</a>=C2=A0is the C++ version and a nice visualisation of it&#39;=
s operators can be found=C2=A0<a href=3D"http://rxmarbles.com/">here</a>.</=
div><div>RxCpp also has experimental support for the Coroutine TS.=C2=A0<a =
href=3D"https://github.com/ReactiveX/RxCpp/blob/master/Rx/v2/examples/await=
able/main.cpp">Checkout</a> their example.</div><div><br></div><div class=
=3D"mt-signature">
        <table border=3D"0" cellpadding=3D"8" cellspacing=3D"0" style=3D"us=
er-select: none;">
            <tbody><tr>
                <td>
                    <a href=3D"https://mailtrack.io?utm_source=3Dgmail&amp;=
utm_medium=3Dsignature&amp;utm_campaign=3Dsignaturevirality5&amp;" class=3D=
"" style=3D"text-decoration:none">
                        <img src=3D"https://s3.amazonaws.com/mailtrack-sign=
ature/sender_notified.gif" alt=3D"Mailtrack" class=3D"" width=3D"32" height=
=3D"32">
                    </a>
                </td>
                <td>
                    <span style=3D"color:#777">Sender notified by</span> <b=
r>
                    <a href=3D"https://mailtrack.io?utm_source=3Dgmail&amp;=
utm_medium=3Dsignature&amp;utm_campaign=3Dsignaturevirality5&amp;" class=3D=
"mt-install" style=3D"color:#4374f7">Mailtrack</a>
                    <span style=3D"color:transparent;font-size:0">11/21/18,=
 11:56:45 PM</span>
                </td>
                <td>
                   =20
                </td>
            </tr>
        </tbody></table>
    </div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr">On Wed, 21 =
Nov 2018 at 18:20, Vu Pham &lt;<a href=3D"mailto:phvu225@gmail.com">phvu225=
@gmail.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=
=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=
=3D"ltr"><div>Hi all,</div><div><br></div><div></div><div>I have this idea =
about a set of primitives for processing elements coming from never-ending =
streams in C++. The main class is <span style=3D"font-family:courier new,mo=
nospace">stream&lt;T&gt;</span> with the following methods:</div><div>=C2=
=A0 - <span style=3D"font-family:courier new,monospace">stream&lt;U&gt; map=
(function&lt;U(T&amp;)&gt; mapper)</span>: map a <span style=3D"font-family=
:courier new,monospace">stream&lt;T&gt;</span> to <span style=3D"font-famil=
y:courier new,monospace">stream&lt;U&gt;</span></div><div>=C2=A0 - <span st=
yle=3D"font-family:courier new,monospace">void foreach(function&lt;void(T&a=
mp;)&gt; processor)</span>: run the given function on each element of the s=
ource stream.</div><div>=C2=A0 - <span style=3D"font-family:courier new,mon=
ospace">void write_to(sink&lt;T&gt;&amp; dest)</span>: write all elements t=
o the given sink.</div><div>=C2=A0 - and so on.</div><div><br></div><div>Al=
l the elements are processed asynchronously, in parallel, using either an E=
xecutor (coming feature of C++) or <span style=3D"font-family:courier new,m=
onospace">std::async</span>.</div><div>The usage will be something like thi=
s:</div><div><br></div><div><span style=3D"font-family:courier new,monospac=
e">stream&lt;MyData&gt; source(...);</span></div><div><span style=3D"font-f=
amily:courier new,monospace">sink&lt;TransformedData&gt; destination(...);<=
/span></div><div><span style=3D"font-family:courier new,monospace">source.m=
ap(...).filter(...).map(...).write_to(destination);</span></div><div><br></=
div><div>I found these constructs very helpful for my usecase, where I proc=
ess messages coming from a distributed queue - something similar to Apache =
Kafka.</div><div>The logic of processing messages can now be expressed usin=
g a chain of transformations. I don&#39;t need to worry about maintaining a=
 threadpool, fibers, and all that threading operations.<br></div><div><br><=
/div><div>I think it is going to be useful for other usecases as well, espe=
cially when we allow users to create their own sources.</div><div>Is there =
anything similar like that in the standard? Do you think something like tha=
t will be helpful?</div><div><br></div><div>Relately, I also have another d=
esign for parallelizing elements from standard collections (vector, list, m=
ap, ...). <br></div><div>Unlike streams, those elements are countable and s=
tored in-memory, so we can define many other aggreation operations like <sp=
an style=3D"font-family:courier new,monospace">reduce</span>, <span style=
=3D"font-family:courier new,monospace">aggregate</span>...</div><div>Using =
these primitives, user can easily parallelize algorithms using functional c=
onstructs, without maintaining low-level threading constructs.</div><div><b=
r></div><div>I believe this is a useful low-overhead abstraction that will =
make people love C++ even more!<br></div><div><br></div><div>I would love t=
o discuss the details.</div><div><br></div><div>Cheers,</div><div>Vu</div><=
div><br></div><div>P.S. This is my first email to this group. I am followin=
g guides from your website. Please forward it to the right channel if this =
is not the correct place :)<br></div><div><br></div></div>

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org" target=3D"_=
blank">std-proposals+unsubscribe@isocpp.org</a>.<br>
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org" target=3D"_blank">std-proposals@isocpp.org</a>.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/067d8f3c-a700-4a4b-9ca7-6596480b20e9%=
40isocpp.org?utm_medium=3Demail&amp;utm_source=3Dfooter" target=3D"_blank">=
https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/067d8f3c-a700-=
4a4b-9ca7-6596480b20e9%40isocpp.org</a>.<br>
</blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"=
 class=3D"gmail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"l=
tr"><div><div dir=3D"ltr"><span style=3D"font-size:12.8px">Regards,</span><=
div style=3D"font-size:12.8px"><br></div><div style=3D"font-size:12.8px">Ga=
etano Checinski</div><div style=3D"font-size:12.8px"><span style=3D"font-si=
ze:12.8px">Founder of Loopperfect</span><br></div><div style=3D"font-size:1=
2.8px"><span style=3D"font-size:12.8px"><a href=3D"https://loopperfect.com"=
 target=3D"_blank">https://loopperfect.com</a></span></div><div style=3D"fo=
nt-size:12.8px"><span style=3D"font-size:12.8px"><a href=3D"https://buckaro=
o.pm" target=3D"_blank">https://buckaroo.pm</a></span></div></div></div></d=
iv></div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org">std-proposa=
ls+unsubscribe@isocpp.org</a>.<br />
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org">std-proposals@isocpp.org</a>.<br />
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/CAN0NuG5OsBskD-X2m3x%3DpTKJUEd9-WiMxV=
3MKJWRnK9YRnpZGA%40mail.gmail.com?utm_medium=3Demail&utm_source=3Dfooter">h=
ttps://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAN0NuG5OsBskD-=
X2m3x%3DpTKJUEd9-WiMxV3MKJWRnK9YRnpZGA%40mail.gmail.com</a>.<br />

--00000000000069bbe6057b358482--

.

--000000000000390d08057b42c388
Content-Type: text/plain; charset="UTF-8"

Thanks for the pointers. It looks like C++ is getting weirder :)

Since I mentioned 2 different things, let's consider each of them and why
they should be part of the standard library. Sorry in advance since this is
going to be a monologue with lots of (potentially ill-informed) opinions.

The stream<T> construct is a pattern that can be used in a wide variety of
applications, including web services, data analysis (processing logs, for
example), distributed computing systems (especially for people who do
scientific computing)...
People came up with many different implementations for asynchronous
event-driven networking libraries for C++, but they tend to be quite
scattered.
The stream<T> may serve as a lingua-franca for these kinds of systems in
C++.
In term of APIs, the closest examples in other languages I can find are
Apache Beam, Flink, Spark. All of them are for the JVM.

The "parallel collection" construct is an easy way to do things in parallel
without messing around with threads.
You may be thinking our vectors and lists are usually not that big, and may
not be very beneficial to parallelize.
But I think part of the problem is it has never been so easy to parallelize
algorithms on collections, that's why we settle with our for-loops and be
good with it.
Now, this construct will allow us to rethink a lot of collection-based
algorithms. For example, I will be very happy to replace qsort() with a
parallel, ideally non-blocking, version of it.

There are similar ideas in other languages: LINQ has a strong SQL flavor
and heavily extends the language, which may not really be what we want.
Scala has parallel versions for all of its collections, which is great in
term of ease-of-use, but I think maintaining the consistency between the
sequential and parallel version will require extra work.
By giving users a mininal set of abstractions, we don't try to be too
opinionated, while allowing it to be implemented efficiently on different
hardware.

It is hard for me to categorize these constructs. They fall somewhere in
the intersection between parallelism, functional programming, and
distributed system, big data, etc. if I must.

The only concern for me is these constructs don't feel very C++ (like
std::async, for example). The first time I saw std::async, I thought:
right, that's what a C++ dude will do, not an Executor.
I understand it will take a lot of time for something to become a standard,
so I am sharing these ideas so that maybe we can make them more like a C++
construct.
Using pipes like in RxCpp is a curious way to do it.

I come from an engineering background where I tend to build distributed
systems to crunch numbers.
The JVM is great for building distributed systems at ease, but not so great
for crunching numbers.
C++ is great for crunching numbers (with BLAS and friends), but not so
great for building distributed systems (you can build great distributed
system in C++, but it will take a significant effort).
I think that is quite unfortunate, and we can definitely expect more from
C++.


Btw, when I said:

> Unlike streams, those elements are countable and stored in-memory...
>>
> I meant to say "finite" instead of "countable", with the implication that
streams are infinite, although nothing is practically infinite in our
profession :)

--
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEZ%2BHLAXfXdQicXiaNDAr3mcc64tekaYU%3DL54rAzbESJw%40mail.gmail.com.

--000000000000390d08057b42c388
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Thanks for the pointers. It looks like C++ is getting=
 weirder :)</div><div><br></div><div>Since I mentioned 2 different things, =
let&#39;s consider each of them and why they should be part of the standard=
 library. Sorry in advance since this is going to be a monologue with lots =
of (potentially ill-informed) opinions.<br></div><div><br></div><div>The st=
ream&lt;T&gt; construct is a pattern that can be used in a wide variety of =
applications, including web services, data analysis (processing logs, for e=
xample), distributed computing systems (especially for people who do scient=
ific computing)...</div><div>People came up with many different implementat=
ions for asynchronous event-driven networking libraries for C++, but they t=
end to be quite scattered.</div><div>The stream&lt;T&gt; may serve as a lin=
gua-franca for these kinds of systems in C++.</div><div>In term of APIs, th=
e closest examples in other languages I can find are Apache Beam, Flink, Sp=
ark. All of them are for the JVM.<br></div><div><br></div><div>The &quot;pa=
rallel collection&quot; construct is an easy way to do things in parallel w=
ithout messing around with threads.</div><div>You may be thinking our vecto=
rs and lists are usually not that big, and may not be very beneficial to pa=
rallelize. <br></div><div>But I think part of the problem is it has never b=
een so easy to parallelize algorithms on collections, that&#39;s why we set=
tle with our for-loops and be good with it.<br></div><div>Now, this constru=
ct will allow us to rethink a lot of collection-based algorithms. For examp=
le, I will be very happy to replace qsort() with a parallel, ideally non-bl=
ocking, version of it.</div><div><br></div><div>There are similar ideas in =
other languages: LINQ has a strong SQL flavor and heavily extends the langu=
age, which may not really be what we want.</div><div>Scala has parallel ver=
sions for all of its collections, which is great in term of ease-of-use, bu=
t I think maintaining the consistency between the sequential and parallel v=
ersion will require extra work.</div><div>By giving users a mininal set of =
abstractions, we don&#39;t try to be too opinionated, while allowing it to =
be implemented efficiently on different hardware.<br></div><div></div><div>=
<br></div><div>It is hard for me to categorize these constructs. They fall =
somewhere in the intersection between parallelism, functional programming, =
and distributed system, big data, etc. if I must.</div><div><br></div><div>=
The only concern for me is these constructs don&#39;t feel very C++ (like s=
td::async, for example). The first time I saw std::async, I thought: right,=
 that&#39;s what a C++ dude will do, not an Executor.</div><div>I understan=
d it will take a lot of time for something to become a standard, so I am sh=
aring these ideas so that maybe we can make them more like a C++ construct.=
<br></div><div>Using pipes like in RxCpp is a curious way to do it.</div><d=
iv><br></div><div>I come from an engineering background where I tend to bui=
ld distributed systems to crunch numbers.</div><div>The JVM is great for bu=
ilding distributed systems at ease, but not so great for crunching numbers.=
</div><div>C++ is great for crunching numbers (with BLAS and friends), but =
not so great for building distributed systems (you can build great distribu=
ted system in C++, but it will take a significant effort).</div><div>I thin=
k that is quite unfortunate, and we can definitely expect more from C++.<br=
></div><div><div><br></div><div><br></div><div>Btw, when I said:<br></div><=
/div><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"=
margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class=
=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8=
ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>Unlik=
e streams, those elements are countable and stored in-memory...</div></div>=
</blockquote></div></blockquote></div><div>I meant to say &quot;finite&quot=
; instead of &quot;countable&quot;, with the implication that streams are i=
nfinite, although nothing is practically infinite in our profession :)</div=
></div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org">std-proposa=
ls+unsubscribe@isocpp.org</a>.<br />
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org">std-proposals@isocpp.org</a>.<br />
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEZ%2BHLAXfXdQicXiaNDAr3mcc64te=
kaYU%3DL54rAzbESJw%40mail.gmail.com?utm_medium=3Demail&utm_source=3Dfooter"=
>https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEZ%2BH=
LAXfXdQicXiaNDAr3mcc64tekaYU%3DL54rAzbESJw%40mail.gmail.com</a>.<br />

--000000000000390d08057b42c388--

.

--0000000000001efa62057be197bb
Content-Type: text/plain; charset="UTF-8"

I think I had an overdose of caffein and came up with another API to make
this more like C++. The idea is to use the extraction and insertion
operators on streams:

stream<T> source(....)
sink<U> my_sink(...)
function<U(T&)> map_fn = ...            // a map function
function<void(U&)> foreach_fn = ...          // a foreach function

// the API:
source >> mapper(map_fn) >> foreach(foreach_fn);

// intermediate results can be stored:
auto mapped = source >> mapper(map_fn);
mapped >> write(my_sink);

- mapper(), foreach(), write()... are utility boilerplate that turn
std::function<..> into an instance of stream_processor.
- the API source >> mapper(map_fn) is implemented as an overload of
operator>>(stream&
s, stream_processor& processor), that returns another stream or void,
depending on the type of the given stream_processor.

I think this can be a C++ way to construct complicated pipelines (I almost
feel like a C++ fashionista).
The extreme of this is to get rid of mapper(), foreach().. altogether: source
>> map_fn >> foreach_fn;
However it may be confusing for developers.

For parallel collections, the summary operations can be supported similarly:
future<U> result = source >> mapper(map_fn) >> aggregator(<initial value>,
<combiner>, <aggregator>);

What do you think?
If this doesn't look good enough, I will double my caffein consumption and
may come up with something better (or worse, no promise).

Cheers,
Vu




On Thu, Nov 22, 2018 at 3:45 PM Vu Pham <phvu225@gmail.com> wrote:

> Thanks for the pointers. It looks like C++ is getting weirder :)
>
> Since I mentioned 2 different things, let's consider each of them and why
> they should be part of the standard library. Sorry in advance since this is
> going to be a monologue with lots of (potentially ill-informed) opinions.
>
> The stream<T> construct is a pattern that can be used in a wide variety of
> applications, including web services, data analysis (processing logs, for
> example), distributed computing systems (especially for people who do
> scientific computing)...
> People came up with many different implementations for asynchronous
> event-driven networking libraries for C++, but they tend to be quite
> scattered.
> The stream<T> may serve as a lingua-franca for these kinds of systems in
> C++.
> In term of APIs, the closest examples in other languages I can find are
> Apache Beam, Flink, Spark. All of them are for the JVM.
>
> The "parallel collection" construct is an easy way to do things in
> parallel without messing around with threads.
> You may be thinking our vectors and lists are usually not that big, and
> may not be very beneficial to parallelize.
> But I think part of the problem is it has never been so easy to
> parallelize algorithms on collections, that's why we settle with our
> for-loops and be good with it.
> Now, this construct will allow us to rethink a lot of collection-based
> algorithms. For example, I will be very happy to replace qsort() with a
> parallel, ideally non-blocking, version of it.
>
> There are similar ideas in other languages: LINQ has a strong SQL flavor
> and heavily extends the language, which may not really be what we want.
> Scala has parallel versions for all of its collections, which is great in
> term of ease-of-use, but I think maintaining the consistency between the
> sequential and parallel version will require extra work.
> By giving users a mininal set of abstractions, we don't try to be too
> opinionated, while allowing it to be implemented efficiently on different
> hardware.
>
> It is hard for me to categorize these constructs. They fall somewhere in
> the intersection between parallelism, functional programming, and
> distributed system, big data, etc. if I must.
>
> The only concern for me is these constructs don't feel very C++ (like
> std::async, for example). The first time I saw std::async, I thought:
> right, that's what a C++ dude will do, not an Executor.
> I understand it will take a lot of time for something to become a
> standard, so I am sharing these ideas so that maybe we can make them more
> like a C++ construct.
> Using pipes like in RxCpp is a curious way to do it.
>
> I come from an engineering background where I tend to build distributed
> systems to crunch numbers.
> The JVM is great for building distributed systems at ease, but not so
> great for crunching numbers.
> C++ is great for crunching numbers (with BLAS and friends), but not so
> great for building distributed systems (you can build great distributed
> system in C++, but it will take a significant effort).
> I think that is quite unfortunate, and we can definitely expect more from
> C++.
>
>
> Btw, when I said:
>
>> Unlike streams, those elements are countable and stored in-memory...
>>>
>> I meant to say "finite" instead of "countable", with the implication that
> streams are infinite, although nothing is practically infinite in our
> profession :)
>


--
PHAM Hoai Vu
LinkedIn: linkedin.com/in/vuphoai

--
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEjnzNnivG%2BH7dYdefqu%2BcBupy5db_0YduhZpiPrfwRvQ%40mail.gmail.com.

--0000000000001efa62057be197bb
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>I think I had an overdose of caffein and came up with=
 another API to make this more like C++. The idea is to use the extraction =
and insertion operators on streams:</div><div><span style=3D"font-family:mo=
nospace,monospace"><br></span></div><div><span style=3D"font-family:monospa=
ce,monospace">stream&lt;T&gt; source(....)</span></div><div><span style=3D"=
font-family:monospace,monospace">sink&lt;U&gt; my_sink(...)<br></span></div=
><div><span style=3D"font-family:monospace,monospace">function&lt;U(T&amp;)=
&gt; map_fn =3D ...=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 // a map function<br></span></div><div><span style=3D"font-fam=
ily:monospace,monospace">function&lt;void(U&amp;)&gt; foreach_fn =3D ...=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 // a foreach function</=
span></div><div><span style=3D"font-family:monospace,monospace"><br></span>=
</div><div><span style=3D"font-family:monospace,monospace">// the API:</spa=
n></div><div><span style=3D"font-family:monospace,monospace">source &gt;&gt=
; mapper(map_fn) &gt;&gt; foreach(foreach_fn);</span></div><div><span style=
=3D"font-family:monospace,monospace"><br></span></div><div><span style=3D"f=
ont-family:monospace,monospace">// intermediate results can be stored:</spa=
n></div><div><span style=3D"font-family:monospace,monospace">auto mapped =
=3D source &gt;&gt; mapper(map_fn);<br></span></div><div><span style=3D"fon=
t-family:monospace,monospace">mapped &gt;&gt; write(my_sink);</span></div><=
div><br></div><div>- <span style=3D"font-family:monospace,monospace">mapper=
(), foreach(), write().</span>.. are utility boilerplate that turn<span sty=
le=3D"font-family:monospace,monospace"> std::function&lt;..&gt;</span> into=
 an instance of <span style=3D"font-family:monospace,monospace">stream_proc=
essor</span>.</div><div>- the API <span style=3D"font-family:monospace,mono=
space">source &gt;&gt; mapper(map_fn)</span> is implemented as an overload =
of <span style=3D"font-family:monospace,monospace">operator&gt;&gt;(stream&=
amp; s, stream_processor&amp; processor)</span>, that returns another <span=
 style=3D"font-family:monospace,monospace">stream</span> or <span style=3D"=
font-family:monospace,monospace">void</span>, depending on the type of the =
given <span style=3D"font-family:monospace,monospace">stream_processor</spa=
n>.<br></div><div><br></div><div>I think this can be a C++ way to construct=
 complicated pipelines (I almost feel like a C++ fashionista).</div><div>Th=
e extreme of this is to get rid of <span style=3D"font-family:monospace,mon=
ospace">mapper(), foreach()</span>.. altogether: <span style=3D"font-family=
:monospace,monospace">source &gt;&gt; map_fn &gt;&gt; foreach_fn;</span></d=
iv><div>However it may be confusing for developers.</div><div><br></div><di=
v>For parallel collections, the summary operations can be supported similar=
ly:</div><div><span style=3D"font-family:monospace,monospace">future&lt;U&g=
t; result =3D source &gt;&gt; mapper(map_fn) &gt;&gt; aggregator(&lt;initia=
l value&gt;, &lt;combiner&gt;, &lt;aggregator&gt;);</span></div><div><br></=
div><div>What do you think?</div><div>If this doesn&#39;t look good enough,=
 I will double my caffein consumption and may come up with something better=
 (or worse, no promise).</div><div><br></div><div>Cheers,</div><div>Vu<br><=
/div><div><br></div><div><br></div><div><br></div></div><br><div class=3D"g=
mail_quote"><div dir=3D"ltr">On Thu, Nov 22, 2018 at 3:45 PM Vu Pham &lt;<a=
 href=3D"mailto:phvu225@gmail.com">phvu225@gmail.com</a>&gt; wrote:<br></di=
v><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:=
1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>Thanks for the point=
ers. It looks like C++ is getting weirder :)</div><div><br></div><div>Since=
 I mentioned 2 different things, let&#39;s consider each of them and why th=
ey should be part of the standard library. Sorry in advance since this is g=
oing to be a monologue with lots of (potentially ill-informed) opinions.<br=
></div><div><br></div><div>The stream&lt;T&gt; construct is a pattern that =
can be used in a wide variety of applications, including web services, data=
 analysis (processing logs, for example), distributed computing systems (es=
pecially for people who do scientific computing)...</div><div>People came u=
p with many different implementations for asynchronous event-driven network=
ing libraries for C++, but they tend to be quite scattered.</div><div>The s=
tream&lt;T&gt; may serve as a lingua-franca for these kinds of systems in C=
++.</div><div>In term of APIs, the closest examples in other languages I ca=
n find are Apache Beam, Flink, Spark. All of them are for the JVM.<br></div=
><div><br></div><div>The &quot;parallel collection&quot; construct is an ea=
sy way to do things in parallel without messing around with threads.</div><=
div>You may be thinking our vectors and lists are usually not that big, and=
 may not be very beneficial to parallelize. <br></div><div>But I think part=
 of the problem is it has never been so easy to parallelize algorithms on c=
ollections, that&#39;s why we settle with our for-loops and be good with it=
..<br></div><div>Now, this construct will allow us to rethink a lot of colle=
ction-based algorithms. For example, I will be very happy to replace qsort(=
) with a parallel, ideally non-blocking, version of it.</div><div><br></div=
><div>There are similar ideas in other languages: LINQ has a strong SQL fla=
vor and heavily extends the language, which may not really be what we want.=
</div><div>Scala has parallel versions for all of its collections, which is=
 great in term of ease-of-use, but I think maintaining the consistency betw=
een the sequential and parallel version will require extra work.</div><div>=
By giving users a mininal set of abstractions, we don&#39;t try to be too o=
pinionated, while allowing it to be implemented efficiently on different ha=
rdware.<br></div><div></div><div><br></div><div>It is hard for me to catego=
rize these constructs. They fall somewhere in the intersection between para=
llelism, functional programming, and distributed system, big data, etc. if =
I must.</div><div><br></div><div>The only concern for me is these construct=
s don&#39;t feel very C++ (like std::async, for example). The first time I =
saw std::async, I thought: right, that&#39;s what a C++ dude will do, not a=
n Executor.</div><div>I understand it will take a lot of time for something=
 to become a standard, so I am sharing these ideas so that maybe we can mak=
e them more like a C++ construct.<br></div><div>Using pipes like in RxCpp i=
s a curious way to do it.</div><div><br></div><div>I come from an engineeri=
ng background where I tend to build distributed systems to crunch numbers.<=
/div><div>The JVM is great for building distributed systems at ease, but no=
t so great for crunching numbers.</div><div>C++ is great for crunching numb=
ers (with BLAS and friends), but not so great for building distributed syst=
ems (you can build great distributed system in C++, but it will take a sign=
ificant effort).</div><div>I think that is quite unfortunate, and we can de=
finitely expect more from C++.<br></div><div><div><br></div><div><br></div>=
<div>Btw, when I said:<br></div></div><div class=3D"gmail_quote"><blockquot=
e class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc sol=
id;padding-left:1ex"><div class=3D"gmail_quote"><blockquote class=3D"gmail_=
quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1=
ex"><div dir=3D"ltr"><div>Unlike streams, those elements are countable and =
stored in-memory...</div></div></blockquote></div></blockquote></div><div>I=
 meant to say &quot;finite&quot; instead of &quot;countable&quot;, with the=
 implication that streams are infinite, although nothing is practically inf=
inite in our profession :)</div></div>
</blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr" class=3D"g=
mail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><div><d=
iv dir=3D"ltr"><div>PHAM Hoai Vu</div><div>LinkedIn:=C2=A0<a href=3D"http:/=
/linkedin.com/in/vuphoai" target=3D"_blank">linkedin.com/in/vuphoai</a></di=
v></div></div></div></div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org">std-proposa=
ls+unsubscribe@isocpp.org</a>.<br />
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org">std-proposals@isocpp.org</a>.<br />
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEjnzNnivG%2BH7dYdefqu%2BcBupy5=
db_0YduhZpiPrfwRvQ%40mail.gmail.com?utm_medium=3Demail&utm_source=3Dfooter"=
>https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEjnzNn=
ivG%2BH7dYdefqu%2BcBupy5db_0YduhZpiPrfwRvQ%40mail.gmail.com</a>.<br />

--0000000000001efa62057be197bb--

.

<html><head></head><body lang=3D"en-US" style=3D"background-color: rgb(255,=
 255, 255); line-height: initial;">                                        =
                                              <div style=3D"width: 100%; fo=
nt-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif=
; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, =
255, 255);">Who decides the parallelism? The stream?</div><div style=3D"wid=
th: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif=
, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-colo=
r: rgb(255, 255, 255);"><br></div><div style=3D"width: 100%; font-size: ini=
tial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb=
(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">=
Your stream might make a good high level API, but we tend to first standard=
ize lower level APIs (from which you can build higher APIs). Like executers=
 and queues and ranges.</div><div style=3D"width: 100%; font-size: initial;=
 font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, =
73, 125); text-align: initial; background-color: rgb(255, 255, 255);"><br><=
/div><div style=3D"width: 100%; font-size: initial; font-family: Calibri, '=
Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: in=
itial; background-color: rgb(255, 255, 255);">(it may be a mistake that we =
focus so much on low level, but if we don't give tools that offer 100% perf=
ormance for all cases (ie by allowing custom allocators, etc) then people j=
ust write there own anyhow.)</div><div style=3D"width: 100%; font-size: ini=
tial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb=
(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">=
<br></div><div style=3D"width: 100%; font-size: initial; font-family: Calib=
ri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-alig=
n: initial; background-color: rgb(255, 255, 255);">If we were to separate y=
our APIs into parts, what do we end up with?</div><div style=3D"width: 100%=
; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-s=
erif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(2=
55, 255, 255);"><br></div>                                                 =
                                                                           =
         <div style=3D"width: 100%; font-size: initial; font-family: Calibr=
i, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align=
: initial; background-color: rgb(255, 255, 255);"><br style=3D"display:init=
ial"></div>                                                                =
                                                                           =
                                                        <div style=3D"font-=
size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; c=
olor: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255=
, 255);">Sent&nbsp;from&nbsp;my&nbsp;BlackBerry&nbsp;portable&nbsp;Babbage&=
nbsp;Device</div>                                                          =
                                                                           =
                                             <table width=3D"100%" style=3D=
"background-color:white;border-spacing:0px;"> <tbody><tr><td colspan=3D"2" =
style=3D"font-size: initial; text-align: initial; background-color: rgb(255=
, 255, 255);">                           <div style=3D"border-style: solid =
none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; pad=
ding: 3pt 0in 0in; font-family: Tahoma, 'BB Alpha Sans', 'Slate Pro'; font-=
size: 10pt;">  <div><b>From: </b>Vu Pham</div><div><b>Sent: </b>Friday, Nov=
ember 30, 2018 8:15 AM</div><div><b>To: </b>std-proposals@isocpp.org</div><=
div><b>Reply To: </b>std-proposals@isocpp.org</div><div><b>Subject: </b>Re:=
 [std-proposals] Asynchronously processing elements of stream in C++</div><=
/div></td></tr></tbody></table><div style=3D"border-style: solid none none;=
 border-top-color: rgb(186, 188, 209); border-top-width: 1pt; font-size: in=
itial; text-align: initial; background-color: rgb(255, 255, 255);"></div><b=
r><div id=3D"_originalContent" style=3D""><div dir=3D"ltr"><div>I think I h=
ad an overdose of caffein and came up with another API to make this more li=
ke C++. The idea is to use the extraction and insertion operators on stream=
s:</div><div><span style=3D"font-family:monospace,monospace"><br></span></d=
iv><div><span style=3D"font-family:monospace,monospace">stream&lt;T&gt; sou=
rce(....)</span></div><div><span style=3D"font-family:monospace,monospace">=
sink&lt;U&gt; my_sink(...)<br></span></div><div><span style=3D"font-family:=
monospace,monospace">function&lt;U(T&amp;)&gt; map_fn =3D ...&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // a map function<br><=
/span></div><div><span style=3D"font-family:monospace,monospace">function&l=
t;void(U&amp;)&gt; foreach_fn =3D ...&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp; // a foreach function</span></div><div><span style=3D"font=
-family:monospace,monospace"><br></span></div><div><span style=3D"font-fami=
ly:monospace,monospace">// the API:</span></div><div><span style=3D"font-fa=
mily:monospace,monospace">source &gt;&gt; mapper(map_fn) &gt;&gt; foreach(f=
oreach_fn);</span></div><div><span style=3D"font-family:monospace,monospace=
"><br></span></div><div><span style=3D"font-family:monospace,monospace">// =
intermediate results can be stored:</span></div><div><span style=3D"font-fa=
mily:monospace,monospace">auto mapped =3D source &gt;&gt; mapper(map_fn);<b=
r></span></div><div><span style=3D"font-family:monospace,monospace">mapped =
&gt;&gt; write(my_sink);</span></div><div><br></div><div>- <span style=3D"f=
ont-family:monospace,monospace">mapper(), foreach(), write().</span>.. are =
utility boilerplate that turn<span style=3D"font-family:monospace,monospace=
"> std::function&lt;..&gt;</span> into an instance of <span style=3D"font-f=
amily:monospace,monospace">stream_processor</span>.</div><div>- the API <sp=
an style=3D"font-family:monospace,monospace">source &gt;&gt; mapper(map_fn)=
</span> is implemented as an overload of <span style=3D"font-family:monospa=
ce,monospace">operator&gt;&gt;(stream&amp; s, stream_processor&amp; process=
or)</span>, that returns another <span style=3D"font-family:monospace,monos=
pace">stream</span> or <span style=3D"font-family:monospace,monospace">void=
</span>, depending on the type of the given <span style=3D"font-family:mono=
space,monospace">stream_processor</span>.<br></div><div><br></div><div>I th=
ink this can be a C++ way to construct complicated pipelines (I almost feel=
 like a C++ fashionista).</div><div>The extreme of this is to get rid of <s=
pan style=3D"font-family:monospace,monospace">mapper(), foreach()</span>.. =
altogether: <span style=3D"font-family:monospace,monospace">source &gt;&gt;=
 map_fn &gt;&gt; foreach_fn;</span></div><div>However it may be confusing f=
or developers.</div><div><br></div><div>For parallel collections, the summa=
ry operations can be supported similarly:</div><div><span style=3D"font-fam=
ily:monospace,monospace">future&lt;U&gt; result =3D source &gt;&gt; mapper(=
map_fn) &gt;&gt; aggregator(&lt;initial value&gt;, &lt;combiner&gt;, &lt;ag=
gregator&gt;);</span></div><div><br></div><div>What do you think?</div><div=
>If this doesn't look good enough, I will double my caffein consumption and=
 may come up with something better (or worse, no promise).</div><div><br></=
div><div>Cheers,</div><div>Vu<br></div><div><br></div><div><br></div><div><=
br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr">On Thu, Nov =
22, 2018 at 3:45 PM Vu Pham &lt;<a href=3D"mailto:phvu225@gmail.com">phvu22=
5@gmail.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=
=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=
=3D"ltr"><div>Thanks for the pointers. It looks like C++ is getting weirder=
 :)</div><div><br></div><div>Since I mentioned 2 different things, let's co=
nsider each of them and why they should be part of the standard library. So=
rry in advance since this is going to be a monologue with lots of (potentia=
lly ill-informed) opinions.<br></div><div><br></div><div>The stream&lt;T&gt=
; construct is a pattern that can be used in a wide variety of applications=
, including web services, data analysis (processing logs, for example), dis=
tributed computing systems (especially for people who do scientific computi=
ng)...</div><div>People came up with many different implementations for asy=
nchronous event-driven networking libraries for C++, but they tend to be qu=
ite scattered.</div><div>The stream&lt;T&gt; may serve as a lingua-franca f=
or these kinds of systems in C++.</div><div>In term of APIs, the closest ex=
amples in other languages I can find are Apache Beam, Flink, Spark. All of =
them are for the JVM.<br></div><div><br></div><div>The "parallel collection=
" construct is an easy way to do things in parallel without messing around =
with threads.</div><div>You may be thinking our vectors and lists are usual=
ly not that big, and may not be very beneficial to parallelize. <br></div><=
div>But I think part of the problem is it has never been so easy to paralle=
lize algorithms on collections, that's why we settle with our for-loops and=
 be good with it.<br></div><div>Now, this construct will allow us to rethin=
k a lot of collection-based algorithms. For example, I will be very happy t=
o replace qsort() with a parallel, ideally non-blocking, version of it.</di=
v><div><br></div><div>There are similar ideas in other languages: LINQ has =
a strong SQL flavor and heavily extends the language, which may not really =
be what we want.</div><div>Scala has parallel versions for all of its colle=
ctions, which is great in term of ease-of-use, but I think maintaining the =
consistency between the sequential and parallel version will require extra =
work.</div><div>By giving users a mininal set of abstractions, we don't try=
 to be too opinionated, while allowing it to be implemented efficiently on =
different hardware.<br></div><div></div><div><br></div><div>It is hard for =
me to categorize these constructs. They fall somewhere in the intersection =
between parallelism, functional programming, and distributed system, big da=
ta, etc. if I must.</div><div><br></div><div>The only concern for me is the=
se constructs don't feel very C++ (like std::async, for example). The first=
 time I saw std::async, I thought: right, that's what a C++ dude will do, n=
ot an Executor.</div><div>I understand it will take a lot of time for somet=
hing to become a standard, so I am sharing these ideas so that maybe we can=
 make them more like a C++ construct.<br></div><div>Using pipes like in RxC=
pp is a curious way to do it.</div><div><br></div><div>I come from an engin=
eering background where I tend to build distributed systems to crunch numbe=
rs.</div><div>The JVM is great for building distributed systems at ease, bu=
t not so great for crunching numbers.</div><div>C++ is great for crunching =
numbers (with BLAS and friends), but not so great for building distributed =
systems (you can build great distributed system in C++, but it will take a =
significant effort).</div><div>I think that is quite unfortunate, and we ca=
n definitely expect more from C++.<br></div><div><div><br></div><div><br></=
div><div>Btw, when I said:<br></div></div><div class=3D"gmail_quote"><block=
quote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc=
 solid;padding-left:1ex"><div class=3D"gmail_quote"><blockquote class=3D"gm=
ail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-le=
ft:1ex"><div dir=3D"ltr"><div>Unlike streams, those elements are countable =
and stored in-memory...</div></div></blockquote></div></blockquote></div><d=
iv>I meant to say "finite" instead of "countable", with the implication tha=
t streams are infinite, although nothing is practically infinite in our pro=
fession :)</div></div>
</blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr" class=3D"g=
mail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><div><d=
iv dir=3D"ltr"><div>PHAM Hoai Vu</div><div>LinkedIn:&nbsp;<a href=3D"http:/=
/linkedin.com/in/vuphoai" target=3D"_blank">linkedin.com/in/vuphoai</a></di=
v></div></div></div></div>

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups "=
ISO C++ Standard - Future Proposals" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org">std-proposa=
ls+unsubscribe@isocpp.org</a>.<br>
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org">std-proposals@isocpp.org</a>.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEjnzNnivG%2BH7dYdefqu%2BcBupy5=
db_0YduhZpiPrfwRvQ%40mail.gmail.com?utm_medium=3Demail&amp;utm_source=3Dfoo=
ter">https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEj=
nzNnivG%2BH7dYdefqu%2BcBupy5db_0YduhZpiPrfwRvQ%40mail.gmail.com</a>.<br>
<br><!--end of _originalContent --></div></body></html>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org">std-proposa=
ls+unsubscribe@isocpp.org</a>.<br />
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org">std-proposals@isocpp.org</a>.<br />
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/20181130154025.5181497.36363.67131%40=
gmail.com?utm_medium=3Demail&utm_source=3Dfooter">https://groups.google.com=
/a/isocpp.org/d/msgid/std-proposals/20181130154025.5181497.36363.67131%40gm=
ail.com</a>.<br />

.

--0000000000008362b1057be4bdd5
Content-Type: text/plain; charset="UTF-8"

Good question. I was imagining the level of parallelization is decided
by a *stream
system,* which serves as both the Scheduler and Factory for the stream
source. Something like this:

stream_system system;
stream<U> source = system.my_source(....);

Another possibility is to add a few utility functions that can be injected
into the >> operators:
source >> mapper(map_fn) >> parallelize(5) >> ...
Much like setprecision()
<http://www.cplusplus.com/reference/iomanip/setprecision/> for std::cout.

Relately, how the parallelism is decided in std::async?

If there is enough interest, I can consolidate this into a concrete
proposal doc.

On Fri, Nov 30, 2018 at 3:40 PM Tony V E <tvaneerd@gmail.com> wrote:

> Who decides the parallelism? The stream?
>
> Your stream might make a good high level API, but we tend to first
> standardize lower level APIs (from which you can build higher APIs). Like
> executers and queues and ranges.
>
> (it may be a mistake that we focus so much on low level, but if we don't
> give tools that offer 100% performance for all cases (ie by allowing custom
> allocators, etc) then people just write there own anyhow.)
>
> If we were to separate your APIs into parts, what do we end up with?
>
>
> Sent from my BlackBerry portable Babbage Device
> *From: *Vu Pham
> *Sent: *Friday, November 30, 2018 8:15 AM
> *To: *std-proposals@isocpp.org
> *Reply To: *std-proposals@isocpp.org
> *Subject: *Re: [std-proposals] Asynchronously processing elements of
> stream in C++
>
> I think I had an overdose of caffein and came up with another API to make
> this more like C++. The idea is to use the extraction and insertion
> operators on streams:
>
> stream<T> source(....)
> sink<U> my_sink(...)
> function<U(T&)> map_fn = ...            // a map function
> function<void(U&)> foreach_fn = ...          // a foreach function
>
> // the API:
> source >> mapper(map_fn) >> foreach(foreach_fn);
>
> // intermediate results can be stored:
> auto mapped = source >> mapper(map_fn);
> mapped >> write(my_sink);
>
> - mapper(), foreach(), write()... are utility boilerplate that turn
> std::function<..> into an instance of stream_processor.
> - the API source >> mapper(map_fn) is implemented as an overload of operator>>(stream&
> s, stream_processor& processor), that returns another stream or void,
> depending on the type of the given stream_processor.
>
> I think this can be a C++ way to construct complicated pipelines (I almost
> feel like a C++ fashionista).
> The extreme of this is to get rid of mapper(), foreach().. altogether: source
> >> map_fn >> foreach_fn;
> However it may be confusing for developers.
>
> For parallel collections, the summary operations can be supported
> similarly:
> future<U> result = source >> mapper(map_fn) >> aggregator(<initial value>,
> <combiner>, <aggregator>);
>
> What do you think?
> If this doesn't look good enough, I will double my caffein consumption and
> may come up with something better (or worse, no promise).
>
> Cheers,
> Vu
>
>
>
>
> On Thu, Nov 22, 2018 at 3:45 PM Vu Pham <phvu225@gmail.com> wrote:
>
>> Thanks for the pointers. It looks like C++ is getting weirder :)
>>
>> Since I mentioned 2 different things, let's consider each of them and why
>> they should be part of the standard library. Sorry in advance since this is
>> going to be a monologue with lots of (potentially ill-informed) opinions.
>>
>> The stream<T> construct is a pattern that can be used in a wide variety
>> of applications, including web services, data analysis (processing logs,
>> for example), distributed computing systems (especially for people who do
>> scientific computing)...
>> People came up with many different implementations for asynchronous
>> event-driven networking libraries for C++, but they tend to be quite
>> scattered.
>> The stream<T> may serve as a lingua-franca for these kinds of systems in
>> C++.
>> In term of APIs, the closest examples in other languages I can find are
>> Apache Beam, Flink, Spark. All of them are for the JVM.
>>
>> The "parallel collection" construct is an easy way to do things in
>> parallel without messing around with threads.
>> You may be thinking our vectors and lists are usually not that big, and
>> may not be very beneficial to parallelize.
>> But I think part of the problem is it has never been so easy to
>> parallelize algorithms on collections, that's why we settle with our
>> for-loops and be good with it.
>> Now, this construct will allow us to rethink a lot of collection-based
>> algorithms. For example, I will be very happy to replace qsort() with a
>> parallel, ideally non-blocking, version of it.
>>
>> There are similar ideas in other languages: LINQ has a strong SQL flavor
>> and heavily extends the language, which may not really be what we want.
>> Scala has parallel versions for all of its collections, which is great in
>> term of ease-of-use, but I think maintaining the consistency between the
>> sequential and parallel version will require extra work.
>> By giving users a mininal set of abstractions, we don't try to be too
>> opinionated, while allowing it to be implemented efficiently on different
>> hardware.
>>
>> It is hard for me to categorize these constructs. They fall somewhere in
>> the intersection between parallelism, functional programming, and
>> distributed system, big data, etc. if I must.
>>
>> The only concern for me is these constructs don't feel very C++ (like
>> std::async, for example). The first time I saw std::async, I thought:
>> right, that's what a C++ dude will do, not an Executor.
>> I understand it will take a lot of time for something to become a
>> standard, so I am sharing these ideas so that maybe we can make them more
>> like a C++ construct.
>> Using pipes like in RxCpp is a curious way to do it.
>>
>> I come from an engineering background where I tend to build distributed
>> systems to crunch numbers.
>> The JVM is great for building distributed systems at ease, but not so
>> great for crunching numbers.
>> C++ is great for crunching numbers (with BLAS and friends), but not so
>> great for building distributed systems (you can build great distributed
>> system in C++, but it will take a significant effort).
>> I think that is quite unfortunate, and we can definitely expect more from
>> C++.
>>
>>
>> Btw, when I said:
>>
>>> Unlike streams, those elements are countable and stored in-memory...
>>>>
>>> I meant to say "finite" instead of "countable", with the implication
>> that streams are infinite, although nothing is practically infinite in our
>> profession :)
>>
>
>
> --
> PHAM Hoai Vu
> LinkedIn: linkedin.com/in/vuphoai
>
> --
> You received this message because you are subscribed to the Google Groups
> "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to std-proposals+unsubscribe@isocpp.org.
> To post to this group, send email to std-proposals@isocpp.org.
> To view this discussion on the web visit
> https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEjnzNnivG%2BH7dYdefqu%2BcBupy5db_0YduhZpiPrfwRvQ%40mail.gmail.com
> <https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEjnzNnivG%2BH7dYdefqu%2BcBupy5db_0YduhZpiPrfwRvQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to std-proposals+unsubscribe@isocpp.org.
> To post to this group, send email to std-proposals@isocpp.org.
> To view this discussion on the web visit
> https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/20181130154025.5181497.36363.67131%40gmail.com
> <https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/20181130154025.5181497.36363.67131%40gmail.com?utm_medium=email&utm_source=footer>
> .
>


--
PHAM Hoai Vu
LinkedIn: linkedin.com/in/vuphoai

--
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEM04Ep64osMAMyzXZbs1xKKBvfXQZejzX-Md%2Bquu%2Bg6Q%40mail.gmail.com.

--0000000000008362b1057be4bdd5
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Good question. I was imagining the level of paralleli=
zation is decided by a <i>stream system,</i> which serves as both the Sched=
uler and Factory for the stream source. Something like this:</div><div><br>=
</div><div>stream_system system;</div><div>stream&lt;U&gt; source =3D syste=
m.my_source(....);</div><div><br></div><div>Another possibility is to add a=
 few utility functions that can be injected into the &gt;&gt; operators:</d=
iv><div>source &gt;&gt; mapper(map_fn) &gt;&gt; parallelize(5) &gt;&gt; ...=
</div><div>Much like <a href=3D"http://www.cplusplus.com/reference/iomanip/=
setprecision/">setprecision()</a> for std::cout.<br></div><div><br></div><d=
iv>Relately, how the parallelism is decided in std::async?</div><div><br></=
div><div>If there is enough interest, I can consolidate this into a concret=
e proposal doc.<br></div></div><br><div class=3D"gmail_quote"><div dir=3D"l=
tr">On Fri, Nov 30, 2018 at 3:40 PM Tony V E &lt;<a href=3D"mailto:tvaneerd=
@gmail.com">tvaneerd@gmail.com</a>&gt; wrote:<br></div><blockquote class=3D=
"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding=
-left:1ex"><div lang=3D"en-US" style=3D"background-color:rgb(255,255,255);l=
ine-height:initial">                                                       =
                               <div style=3D"width:100%;font-size:initial;f=
ont-family:Calibri,&#39;Slate Pro&#39;,sans-serif,sans-serif;color:rgb(31,7=
3,125);text-align:initial;background-color:rgb(255,255,255)">Who decides th=
e parallelism? The stream?</div><div style=3D"width:100%;font-size:initial;=
font-family:Calibri,&#39;Slate Pro&#39;,sans-serif,sans-serif;color:rgb(31,=
73,125);text-align:initial;background-color:rgb(255,255,255)"><br></div><di=
v style=3D"width:100%;font-size:initial;font-family:Calibri,&#39;Slate Pro&=
#39;,sans-serif,sans-serif;color:rgb(31,73,125);text-align:initial;backgrou=
nd-color:rgb(255,255,255)">Your stream might make a good high level API, bu=
t we tend to first standardize lower level APIs (from which you can build h=
igher APIs). Like executers and queues and ranges.</div><div style=3D"width=
:100%;font-size:initial;font-family:Calibri,&#39;Slate Pro&#39;,sans-serif,=
sans-serif;color:rgb(31,73,125);text-align:initial;background-color:rgb(255=
,255,255)"><br></div><div style=3D"width:100%;font-size:initial;font-family=
:Calibri,&#39;Slate Pro&#39;,sans-serif,sans-serif;color:rgb(31,73,125);tex=
t-align:initial;background-color:rgb(255,255,255)">(it may be a mistake tha=
t we focus so much on low level, but if we don&#39;t give tools that offer =
100% performance for all cases (ie by allowing custom allocators, etc) then=
 people just write there own anyhow.)</div><div style=3D"width:100%;font-si=
ze:initial;font-family:Calibri,&#39;Slate Pro&#39;,sans-serif,sans-serif;co=
lor:rgb(31,73,125);text-align:initial;background-color:rgb(255,255,255)"><b=
r></div><div style=3D"width:100%;font-size:initial;font-family:Calibri,&#39=
;Slate Pro&#39;,sans-serif,sans-serif;color:rgb(31,73,125);text-align:initi=
al;background-color:rgb(255,255,255)">If we were to separate your APIs into=
 parts, what do we end up with?</div><div style=3D"width:100%;font-size:ini=
tial;font-family:Calibri,&#39;Slate Pro&#39;,sans-serif,sans-serif;color:rg=
b(31,73,125);text-align:initial;background-color:rgb(255,255,255)"><br></di=
v>                                                                         =
                                                            <div style=3D"w=
idth:100%;font-size:initial;font-family:Calibri,&#39;Slate Pro&#39;,sans-se=
rif,sans-serif;color:rgb(31,73,125);text-align:initial;background-color:rgb=
(255,255,255)"><br style=3D"display:initial"></div>                        =
                                                                           =
                                                                           =
                     <div style=3D"font-size:initial;font-family:Calibri,&#=
39;Slate Pro&#39;,sans-serif,sans-serif;color:rgb(31,73,125);text-align:ini=
tial;background-color:rgb(255,255,255)">Sent=C2=A0from=C2=A0my=C2=A0BlackBe=
rry=C2=A0portable=C2=A0Babbage=C2=A0Device</div>                           =
                                                                           =
                                                                           =
 <table width=3D"100%" style=3D"background-color:white;border-spacing:0px">=
 <tbody><tr><td colspan=3D"2" style=3D"font-size:initial;text-align:initial=
;background-color:rgb(255,255,255)">                           <div style=
=3D"border-style:solid none none;border-top-color:rgb(181,196,223);border-t=
op-width:1pt;padding:3pt 0in 0in;font-family:Tahoma,&#39;BB Alpha Sans&#39;=
,&#39;Slate Pro&#39;;font-size:10pt">  <div><b>From: </b>Vu Pham</div><div>=
<b>Sent: </b>Friday, November 30, 2018 8:15 AM</div><div><b>To: </b><a href=
=3D"mailto:std-proposals@isocpp.org" target=3D"_blank">std-proposals@isocpp=
..org</a></div><div><b>Reply To: </b><a href=3D"mailto:std-proposals@isocpp.=
org" target=3D"_blank">std-proposals@isocpp.org</a></div><div><b>Subject: <=
/b>Re: [std-proposals] Asynchronously processing elements of stream in C++<=
/div></div></td></tr></tbody></table><div style=3D"border-style:solid none =
none;border-top-color:rgb(186,188,209);border-top-width:1pt;font-size:initi=
al;text-align:initial;background-color:rgb(255,255,255)"></div><br><div id=
=3D"m_-5368812360860644601_originalContent"><div dir=3D"ltr"><div>I think I=
 had an overdose of caffein and came up with another API to make this more =
like C++. The idea is to use the extraction and insertion operators on stre=
ams:</div><div><span style=3D"font-family:monospace,monospace"><br></span><=
/div><div><span style=3D"font-family:monospace,monospace">stream&lt;T&gt; s=
ource(....)</span></div><div><span style=3D"font-family:monospace,monospace=
">sink&lt;U&gt; my_sink(...)<br></span></div><div><span style=3D"font-famil=
y:monospace,monospace">function&lt;U(T&amp;)&gt; map_fn =3D ...=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 // a map function<br=
></span></div><div><span style=3D"font-family:monospace,monospace">function=
&lt;void(U&amp;)&gt; foreach_fn =3D ...=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 // a foreach function</span></div><div><span style=3D"fo=
nt-family:monospace,monospace"><br></span></div><div><span style=3D"font-fa=
mily:monospace,monospace">// the API:</span></div><div><span style=3D"font-=
family:monospace,monospace">source &gt;&gt; mapper(map_fn) &gt;&gt; foreach=
(foreach_fn);</span></div><div><span style=3D"font-family:monospace,monospa=
ce"><br></span></div><div><span style=3D"font-family:monospace,monospace">/=
/ intermediate results can be stored:</span></div><div><span style=3D"font-=
family:monospace,monospace">auto mapped =3D source &gt;&gt; mapper(map_fn);=
<br></span></div><div><span style=3D"font-family:monospace,monospace">mappe=
d &gt;&gt; write(my_sink);</span></div><div><br></div><div>- <span style=3D=
"font-family:monospace,monospace">mapper(), foreach(), write().</span>.. ar=
e utility boilerplate that turn<span style=3D"font-family:monospace,monospa=
ce"> std::function&lt;..&gt;</span> into an instance of <span style=3D"font=
-family:monospace,monospace">stream_processor</span>.</div><div>- the API <=
span style=3D"font-family:monospace,monospace">source &gt;&gt; mapper(map_f=
n)</span> is implemented as an overload of <span style=3D"font-family:monos=
pace,monospace">operator&gt;&gt;(stream&amp; s, stream_processor&amp; proce=
ssor)</span>, that returns another <span style=3D"font-family:monospace,mon=
ospace">stream</span> or <span style=3D"font-family:monospace,monospace">vo=
id</span>, depending on the type of the given <span style=3D"font-family:mo=
nospace,monospace">stream_processor</span>.<br></div><div><br></div><div>I =
think this can be a C++ way to construct complicated pipelines (I almost fe=
el like a C++ fashionista).</div><div>The extreme of this is to get rid of =
<span style=3D"font-family:monospace,monospace">mapper(), foreach()</span>.=
.. altogether: <span style=3D"font-family:monospace,monospace">source &gt;&g=
t; map_fn &gt;&gt; foreach_fn;</span></div><div>However it may be confusing=
 for developers.</div><div><br></div><div>For parallel collections, the sum=
mary operations can be supported similarly:</div><div><span style=3D"font-f=
amily:monospace,monospace">future&lt;U&gt; result =3D source &gt;&gt; mappe=
r(map_fn) &gt;&gt; aggregator(&lt;initial value&gt;, &lt;combiner&gt;, &lt;=
aggregator&gt;);</span></div><div><br></div><div>What do you think?</div><d=
iv>If this doesn&#39;t look good enough, I will double my caffein consumpti=
on and may come up with something better (or worse, no promise).</div><div>=
<br></div><div>Cheers,</div><div>Vu<br></div><div><br></div><div><br></div>=
<div><br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr">On Thu=
, Nov 22, 2018 at 3:45 PM Vu Pham &lt;<a href=3D"mailto:phvu225@gmail.com" =
target=3D"_blank">phvu225@gmail.com</a>&gt; wrote:<br></div><blockquote cla=
ss=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pa=
dding-left:1ex"><div dir=3D"ltr"><div>Thanks for the pointers. It looks lik=
e C++ is getting weirder :)</div><div><br></div><div>Since I mentioned 2 di=
fferent things, let&#39;s consider each of them and why they should be part=
 of the standard library. Sorry in advance since this is going to be a mono=
logue with lots of (potentially ill-informed) opinions.<br></div><div><br><=
/div><div>The stream&lt;T&gt; construct is a pattern that can be used in a =
wide variety of applications, including web services, data analysis (proces=
sing logs, for example), distributed computing systems (especially for peop=
le who do scientific computing)...</div><div>People came up with many diffe=
rent implementations for asynchronous event-driven networking libraries for=
 C++, but they tend to be quite scattered.</div><div>The stream&lt;T&gt; ma=
y serve as a lingua-franca for these kinds of systems in C++.</div><div>In =
term of APIs, the closest examples in other languages I can find are Apache=
 Beam, Flink, Spark. All of them are for the JVM.<br></div><div><br></div><=
div>The &quot;parallel collection&quot; construct is an easy way to do thin=
gs in parallel without messing around with threads.</div><div>You may be th=
inking our vectors and lists are usually not that big, and may not be very =
beneficial to parallelize. <br></div><div>But I think part of the problem i=
s it has never been so easy to parallelize algorithms on collections, that&=
#39;s why we settle with our for-loops and be good with it.<br></div><div>N=
ow, this construct will allow us to rethink a lot of collection-based algor=
ithms. For example, I will be very happy to replace qsort() with a parallel=
, ideally non-blocking, version of it.</div><div><br></div><div>There are s=
imilar ideas in other languages: LINQ has a strong SQL flavor and heavily e=
xtends the language, which may not really be what we want.</div><div>Scala =
has parallel versions for all of its collections, which is great in term of=
 ease-of-use, but I think maintaining the consistency between the sequentia=
l and parallel version will require extra work.</div><div>By giving users a=
 mininal set of abstractions, we don&#39;t try to be too opinionated, while=
 allowing it to be implemented efficiently on different hardware.<br></div>=
<div></div><div><br></div><div>It is hard for me to categorize these constr=
ucts. They fall somewhere in the intersection between parallelism, function=
al programming, and distributed system, big data, etc. if I must.</div><div=
><br></div><div>The only concern for me is these constructs don&#39;t feel =
very C++ (like std::async, for example). The first time I saw std::async, I=
 thought: right, that&#39;s what a C++ dude will do, not an Executor.</div>=
<div>I understand it will take a lot of time for something to become a stan=
dard, so I am sharing these ideas so that maybe we can make them more like =
a C++ construct.<br></div><div>Using pipes like in RxCpp is a curious way t=
o do it.</div><div><br></div><div>I come from an engineering background whe=
re I tend to build distributed systems to crunch numbers.</div><div>The JVM=
 is great for building distributed systems at ease, but not so great for cr=
unching numbers.</div><div>C++ is great for crunching numbers (with BLAS an=
d friends), but not so great for building distributed systems (you can buil=
d great distributed system in C++, but it will take a significant effort).<=
/div><div>I think that is quite unfortunate, and we can definitely expect m=
ore from C++.<br></div><div><div><br></div><div><br></div><div>Btw, when I =
said:<br></div></div><div class=3D"gmail_quote"><blockquote class=3D"gmail_=
quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1=
ex"><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"m=
argin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"l=
tr"><div>Unlike streams, those elements are countable and stored in-memory.=
...</div></div></blockquote></div></blockquote></div><div>I meant to say &qu=
ot;finite&quot; instead of &quot;countable&quot;, with the implication that=
 streams are infinite, although nothing is practically infinite in our prof=
ession :)</div></div>
</blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr" class=3D"m=
_-5368812360860644601gmail_signature" data-smartmail=3D"gmail_signature"><d=
iv dir=3D"ltr"><div><div dir=3D"ltr"><div>PHAM Hoai Vu</div><div>LinkedIn:=
=C2=A0<a href=3D"http://linkedin.com/in/vuphoai" target=3D"_blank">linkedin=
..com/in/vuphoai</a></div></div></div></div></div>

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org" target=3D"_=
blank">std-proposals+unsubscribe@isocpp.org</a>.<br>
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org" target=3D"_blank">std-proposals@isocpp.org</a>.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEjnzNnivG%2BH7dYdefqu%2BcBupy5=
db_0YduhZpiPrfwRvQ%40mail.gmail.com?utm_medium=3Demail&amp;utm_source=3Dfoo=
ter" target=3D"_blank">https://groups.google.com/a/isocpp.org/d/msgid/std-p=
roposals/CAFhKpXEjnzNnivG%2BH7dYdefqu%2BcBupy5db_0YduhZpiPrfwRvQ%40mail.gma=
il.com</a>.<br>
<br></div></div>

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org" target=3D"_=
blank">std-proposals+unsubscribe@isocpp.org</a>.<br>
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org" target=3D"_blank">std-proposals@isocpp.org</a>.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/20181130154025.5181497.36363.67131%40=
gmail.com?utm_medium=3Demail&amp;utm_source=3Dfooter" target=3D"_blank">htt=
ps://groups.google.com/a/isocpp.org/d/msgid/std-proposals/20181130154025.51=
81497.36363.67131%40gmail.com</a>.<br>
</blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr" class=3D"g=
mail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><div><d=
iv dir=3D"ltr"><div>PHAM Hoai Vu</div><div>LinkedIn:=C2=A0<a href=3D"http:/=
/linkedin.com/in/vuphoai" target=3D"_blank">linkedin.com/in/vuphoai</a></di=
v></div></div></div></div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org">std-proposa=
ls+unsubscribe@isocpp.org</a>.<br />
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org">std-proposals@isocpp.org</a>.<br />
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEM04Ep64osMAMyzXZbs1xKKBvfXQZe=
jzX-Md%2Bquu%2Bg6Q%40mail.gmail.com?utm_medium=3Demail&utm_source=3Dfooter"=
>https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEM04Ep=
64osMAMyzXZbs1xKKBvfXQZejzX-Md%2Bquu%2Bg6Q%40mail.gmail.com</a>.<br />

--0000000000008362b1057be4bdd5--

.

--00000000000076206e057be54a5a
Content-Type: text/plain; charset="UTF-8"

I'd like to emphasize by repeating my previous statement:

> There are many different ways to implement functional reactive(FRP) data
processing pipelines, each with very different properties.

I think this discussion would benefit by talking about the various
properties your API implicates.

As a site note: I implemented a coroutine and pull based stream processing
library: https://github.com/LoopPerfect/conduit
The API you depicted is very similar to conduits.

Here a list of questions that are important in my opinion:

*Push or Pull ? - intersection of both?*

*Examples*:

*Stream<tuple<T, U>> stream3 = zipWithLatest(stream1, stream2)*
*Interval(ms) **// emits a counter and increments every ms*

*stream >> throttle(n, ms) // emits up to n events per ms*


*First order or Higher order FRP ?*

*Example:*


*stream1 >> map([](int ms){ return Interval(ms);  }); *

*counter() >> switchMap({stream1, stream2}, [](int i){ return i%2; });*

*// alternates between elements between stream 1 and stream 2*


*Questions:*
*does the counter fire if noone is listening?*
*Can you subscribe after unsubscribe? - can you suspend and resume the
event-source?*

*Backpressure and Buffering*
*Example:*

counter()
  >> flatMap([](auto x){ return delayAndEmit(10ms, x); })
  >> flatMap([](auto x){ return delayAndEmit(100ms, x); })


*Questions:*

After every emission, 10 more events are generated, are they buffered,
swallowed or delayed via backpressure?
Do you buffer events?
what happens if the buffer is full?
how do you buffer?


*State Management and Purity*


counter()
  >> scan(0, [](int a, int b){ return a+b; }) // emits the partial sum 0...n

In order to make scan work, we need to store it's state somewhere. Where do
we store it?

It might sound like a silly question but bear with me:

What if you want to serialize the state of the eventstream, close the
application and resume the stream after restart?

Transducer based stream processing libraries maintain often the separation
of sinks, producers and pipes(transformers).
The subscriber/sink then receives  a stateless pipe and a associated
stateobject and drives the iteration.
This also enables the end user to decide various details FRP questions like
push vs. pull or in the switchmap example.

*subscriptions: static or dynamic*

Can I create a firing observable and subscribe to it later?
Can you have multiple subscribers? - Will everyone receive the same events?

*Regarding Schedulers and Executors:*

RxCpp has shown that the decision how things are executed can be deferred
by the end user.
I don't see a big problem why your proposed API should not be able to
support this.
However I'm not sure whether offering support for custom schedulers and
executors support will be free (either cost in performance or complexity)


Now the real question: Can we design *one* generic/abstract API that can
defer all those question to the implementer?


[image: Mailtrack]
<https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&>
Sender
notified by
Mailtrack
<https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&>
11/30/18,
5:14:43 PM

On Fri, 30 Nov 2018 at 15:40, Tony V E <tvaneerd@gmail.com> wrote:

> Who decides the parallelism? The stream?
>
> Your stream might make a good high level API, but we tend to first
> standardize lower level APIs (from which you can build higher APIs). Like
> executers and queues and ranges.
>
> (it may be a mistake that we focus so much on low level, but if we don't
> give tools that offer 100% performance for all cases (ie by allowing custom
> allocators, etc) then people just write there own anyhow.)
>
> If we were to separate your APIs into parts, what do we end up with?
>
>
> Sent from my BlackBerry portable Babbage Device
> *From: *Vu Pham
> *Sent: *Friday, November 30, 2018 8:15 AM
> *To: *std-proposals@isocpp.org
> *Reply To: *std-proposals@isocpp.org
> *Subject: *Re: [std-proposals] Asynchronously processing elements of
> stream in C++
>
> I think I had an overdose of caffein and came up with another API to make
> this more like C++. The idea is to use the extraction and insertion
> operators on streams:
>
> stream<T> source(....)
> sink<U> my_sink(...)
> function<U(T&)> map_fn = ...            // a map function
> function<void(U&)> foreach_fn = ...          // a foreach function
>
> // the API:
> source >> mapper(map_fn) >> foreach(foreach_fn);
>
> // intermediate results can be stored:
> auto mapped = source >> mapper(map_fn);
> mapped >> write(my_sink);
>
> - mapper(), foreach(), write()... are utility boilerplate that turn
> std::function<..> into an instance of stream_processor.
> - the API source >> mapper(map_fn) is implemented as an overload of operator>>(stream&
> s, stream_processor& processor), that returns another stream or void,
> depending on the type of the given stream_processor.
>
> I think this can be a C++ way to construct complicated pipelines (I almost
> feel like a C++ fashionista).
> The extreme of this is to get rid of mapper(), foreach().. altogether: source
> >> map_fn >> foreach_fn;
> However it may be confusing for developers.
>
> For parallel collections, the summary operations can be supported
> similarly:
> future<U> result = source >> mapper(map_fn) >> aggregator(<initial value>,
> <combiner>, <aggregator>);
>
> What do you think?
> If this doesn't look good enough, I will double my caffein consumption and
> may come up with something better (or worse, no promise).
>
> Cheers,
> Vu
>
>
>
>
> On Thu, Nov 22, 2018 at 3:45 PM Vu Pham <phvu225@gmail.com> wrote:
>
>> Thanks for the pointers. It looks like C++ is getting weirder :)
>>
>> Since I mentioned 2 different things, let's consider each of them and why
>> they should be part of the standard library. Sorry in advance since this is
>> going to be a monologue with lots of (potentially ill-informed) opinions.
>>
>> The stream<T> construct is a pattern that can be used in a wide variety
>> of applications, including web services, data analysis (processing logs,
>> for example), distributed computing systems (especially for people who do
>> scientific computing)...
>> People came up with many different implementations for asynchronous
>> event-driven networking libraries for C++, but they tend to be quite
>> scattered.
>> The stream<T> may serve as a lingua-franca for these kinds of systems in
>> C++.
>> In term of APIs, the closest examples in other languages I can find are
>> Apache Beam, Flink, Spark. All of them are for the JVM.
>>
>> The "parallel collection" construct is an easy way to do things in
>> parallel without messing around with threads.
>> You may be thinking our vectors and lists are usually not that big, and
>> may not be very beneficial to parallelize.
>> But I think part of the problem is it has never been so easy to
>> parallelize algorithms on collections, that's why we settle with our
>> for-loops and be good with it.
>> Now, this construct will allow us to rethink a lot of collection-based
>> algorithms. For example, I will be very happy to replace qsort() with a
>> parallel, ideally non-blocking, version of it.
>>
>> There are similar ideas in other languages: LINQ has a strong SQL flavor
>> and heavily extends the language, which may not really be what we want.
>> Scala has parallel versions for all of its collections, which is great in
>> term of ease-of-use, but I think maintaining the consistency between the
>> sequential and parallel version will require extra work.
>> By giving users a mininal set of abstractions, we don't try to be too
>> opinionated, while allowing it to be implemented efficiently on different
>> hardware.
>>
>> It is hard for me to categorize these constructs. They fall somewhere in
>> the intersection between parallelism, functional programming, and
>> distributed system, big data, etc. if I must.
>>
>> The only concern for me is these constructs don't feel very C++ (like
>> std::async, for example). The first time I saw std::async, I thought:
>> right, that's what a C++ dude will do, not an Executor.
>> I understand it will take a lot of time for something to become a
>> standard, so I am sharing these ideas so that maybe we can make them more
>> like a C++ construct.
>> Using pipes like in RxCpp is a curious way to do it.
>>
>> I come from an engineering background where I tend to build distributed
>> systems to crunch numbers.
>> The JVM is great for building distributed systems at ease, but not so
>> great for crunching numbers.
>> C++ is great for crunching numbers (with BLAS and friends), but not so
>> great for building distributed systems (you can build great distributed
>> system in C++, but it will take a significant effort).
>> I think that is quite unfortunate, and we can definitely expect more from
>> C++.
>>
>>
>> Btw, when I said:
>>
>>> Unlike streams, those elements are countable and stored in-memory...
>>>>
>>> I meant to say "finite" instead of "countable", with the implication
>> that streams are infinite, although nothing is practically infinite in our
>> profession :)
>>
>
>
> --
> PHAM Hoai Vu
> LinkedIn: linkedin.com/in/vuphoai
>
> --
> You received this message because you are subscribed to the Google Groups
> "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to std-proposals+unsubscribe@isocpp.org.
> To post to this group, send email to std-proposals@isocpp.org.
> To view this discussion on the web visit
> https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEjnzNnivG%2BH7dYdefqu%2BcBupy5db_0YduhZpiPrfwRvQ%40mail.gmail.com
> <https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEjnzNnivG%2BH7dYdefqu%2BcBupy5db_0YduhZpiPrfwRvQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to std-proposals+unsubscribe@isocpp.org.
> To post to this group, send email to std-proposals@isocpp.org.
> To view this discussion on the web visit
> https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/20181130154025.5181497.36363.67131%40gmail.com
> <https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/20181130154025.5181497.36363.67131%40gmail.com?utm_medium=email&utm_source=footer>
> .
>


--
Regards,

Gaetano Checinski
Founder of Loopperfect
https://loopperfect.com
https://buckaroo.pm

--
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAN0NuG761OD5pMWemQJW2-OqhqUTf1Gq8G795uOwtmdP6khWRQ%40mail.gmail.com.

--00000000000076206e057be54a5a
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><img width=3D"0" height=3D"0" class=3D"mailtrack-img" alt=
=3D"" style=3D"display:flex" src=3D"https://mailtrack.io/trace/mail/1d37e28=
bdd0ff7cbe9f5fd6a99b35e1bc7079743.png?u=3D931501"><div></div><div dir=3D"lt=
r"><div></div><div>I&#39;d like to emphasize by repeating my previous state=
ment:<br></div><div><br>&gt; There are many different ways to implement fun=
ctional reactive(FRP) data processing pipelines, each with very different p=
roperties.<font color=3D"#1f497d" face=3D"Calibri, Slate Pro, sans-serif, s=
ans-serif" size=3D"3"><br></font><br>I think this discussion would benefit =
by talking about the various properties your API implicates.</div><div><br>=
</div><div>As a site note: I implemented a coroutine and pull based stream =
processing library:=C2=A0<a href=3D"https://github.com/LoopPerfect/conduit"=
 target=3D"_blank">https://github.com/LoopPerfect/conduit</a><br>The API yo=
u depicted is very similar to conduits.</div><div><br></div><div>Here a lis=
t of questions that are important in my opinion:</div><div><br></div><div><=
b>Push or Pull ? - intersection of both?</b></div><div><br></div><div><b>Ex=
amples</b>:=C2=A0</div><div><br></div></div><blockquote style=3D"margin:0 0=
 0 40px;border:none;padding:0px"><div dir=3D"ltr"><div><font face=3D"monosp=
ace, monospace"><i>Stream&lt;tuple&lt;T, U&gt;&gt; stream3 =3D zipWithLates=
t(stream1, stream2)</i></font></div><div><i style=3D"font-family:monospace,=
monospace">Interval(ms)=C2=A0</i><i style=3D"font-family:monospace,monospac=
e">// emits a counter and increments every ms</i><font face=3D"monospace, m=
onospace"><i><br></i></font></div></div></blockquote><blockquote style=3D"m=
argin:0 0 0 40px;border:none;padding:0px"><div dir=3D"ltr"><div><i style=3D=
"font-family:monospace,monospace">stream &gt;&gt; throttle(n, ms) // emits =
up to n events per ms</i></div><div><i style=3D"font-family:monospace,monos=
pace"><br></i></div></div></blockquote><div style=3D"font-style:italic;font=
-family:monospace,monospace"><i style=3D"font-family:monospace,monospace"><=
br></i></div><div><font face=3D"arial, helvetica, sans-serif"><b>First orde=
r or Higher order FRP ?</b></font></div><div><font face=3D"arial, helvetica=
, sans-serif"><b><br></b></font></div><div><font face=3D"arial, helvetica, =
sans-serif"><b>Example:</b></font></div><font face=3D"monospace, monospace"=
 style=3D"font-style:italic">=C2=A0</font><br><blockquote style=3D"margin:0=
 0 0 40px;border:none;padding:0px"><div dir=3D"ltr"><div><font face=3D"mono=
space, monospace"><i>stream1 &gt;&gt; map([](int ms){ return Interval(ms);=
=C2=A0 });=C2=A0</i></font></div></div></blockquote><blockquote style=3D"ma=
rgin:0 0 0 40px;border:none;padding:0px"><div dir=3D"ltr"><div><font face=
=3D"monospace, monospace"><i>counter() &gt;&gt; switchMap({stream1, stream2=
}, [](int i){ return i%2; });</i></font></div></div></blockquote><blockquot=
e style=3D"margin:0 0 0 40px;border:none;padding:0px"><div dir=3D"ltr"><div=
><font face=3D"monospace, monospace"><i>// alternates between elements betw=
een stream 1 and stream 2</i></font></div></div></blockquote><blockquote st=
yle=3D"margin:0 0 0 40px;border:none;padding:0px"><div dir=3D"ltr"><div><i =
style=3D"font-family:monospace,monospace"><br></i></div></div></blockquote>=
<i><font face=3D"arial, helvetica, sans-serif">Questions:</font></i><div><i=
><font face=3D"arial, helvetica, sans-serif">does the counter fire if noone=
 is listening?</font></i></div><div><i><font face=3D"arial, helvetica, sans=
-serif">Can you subscribe after unsubscribe? - can you suspend and resume t=
he event-source?</font></i></div><div><br></div><div><font face=3D"arial, h=
elvetica, sans-serif"><i><b>Backpressure and Buffering</b></i></font></div>=
<div><font face=3D"arial, helvetica, sans-serif"><i><b>Example:</b></i></fo=
nt></div><div><font face=3D"arial, helvetica, sans-serif"><i><b><br></b></i=
></font></div><blockquote style=3D"margin:0 0 0 40px;border:none;padding:0p=
x"><div><font face=3D"monospace, monospace">counter()=C2=A0</font></div><di=
v><font face=3D"monospace, monospace">=C2=A0 &gt;&gt; flatMap([](auto x){ r=
eturn delayAndEmit(10ms, x); })</font></div><div><font face=3D"monospace, m=
onospace">=C2=A0 &gt;&gt; flatMap([](auto x){ return delayAndEmit(100ms, x)=
; })=C2=A0</font></div></blockquote><div><br></div><div><font face=3D"arial=
, helvetica, sans-serif"><b><i>Questions:</i></b></font></div><div><font fa=
ce=3D"arial, helvetica, sans-serif"><br></font></div><div><font face=3D"ari=
al, helvetica, sans-serif">After every emission, 10 more events are generat=
ed, are they buffered, swallowed or delayed via backpressure?=C2=A0</font><=
br></div><div><font face=3D"arial, helvetica, sans-serif">Do you buffer eve=
nts?=C2=A0</font></div><div><font face=3D"arial, helvetica, sans-serif">wha=
t happens if the buffer is full?=C2=A0=C2=A0</font></div><div><font face=3D=
"arial, helvetica, sans-serif">how do you buffer?</font></div><div><font fa=
ce=3D"arial, helvetica, sans-serif"><br></font></div><div><font face=3D"ari=
al, helvetica, sans-serif"><br></font></div><div><font face=3D"arial, helve=
tica, sans-serif"><b>State Management=C2=A0and Purity</b></font></div><bloc=
kquote style=3D"margin:0 0 0 40px;border:none;padding:0px"><br></blockquote=
><blockquote style=3D"margin:0 0 0 40px;border:none;padding:0px"><div><div>=
<font face=3D"monospace, monospace">counter()=C2=A0</font></div></div><div>=
<div><font face=3D"monospace, monospace">=C2=A0 &gt;&gt; scan(0, [](int a, =
int b){ return a+b; }) // emits the partial sum 0...n</font></div></div><di=
v><br></div></blockquote>In order to make scan work, we need to store it&#3=
9;s state somewhere. Where do we store it?<div><br></div><div>It might soun=
d like a silly question but bear with me:</div><div><br></div><div>What if =
you want to serialize the state of the eventstream, close the application a=
nd resume the stream after restart?</div><div><br></div><div>Transducer bas=
ed stream processing libraries maintain often the separation of sinks, prod=
ucers and pipes(transformers).</div><div>The subscriber/sink then receives=
=C2=A0 a stateless pipe and a associated stateobject and drives the iterati=
on.=C2=A0</div><div>This also enables the end user to decide various detail=
s FRP questions like push vs. pull or in the switchmap example.</div><div><=
br></div><div><div><div dir=3D"ltr"><div><b>subscriptions: static or dynami=
c</b></div><div><br></div><div>Can I create a firing observable and subscri=
be to it later?</div><div>Can you have multiple subscribers? - Will everyon=
e receive the same events?</div><div><br></div><div><b>Regarding Schedulers=
 and Executors:</b></div><div><br></div><div>RxCpp has shown that the decis=
ion how things are executed can be deferred by the end user.=C2=A0</div><di=
v>I don&#39;t see a big problem why your proposed API should not be able to=
 support this.=C2=A0</div><div>However I&#39;m not sure whether offering su=
pport for custom schedulers and executors support will be free (either cost=
 in performance or complexity)</div><br></div><br></div><div>Now the real q=
uestion: Can we design <b>one</b> generic/abstract API that can defer all t=
hose question to the implementer?=C2=A0</div><br></div><br><div class=3D"mt=
-signature">
        <table border=3D"0" cellpadding=3D"8" cellspacing=3D"0" style=3D"us=
er-select: none;">
            <tbody><tr>
                <td>
                    <a href=3D"https://mailtrack.io?utm_source=3Dgmail&amp;=
utm_medium=3Dsignature&amp;utm_campaign=3Dsignaturevirality5&amp;" class=3D=
"" style=3D"text-decoration:none">
                        <img src=3D"https://s3.amazonaws.com/mailtrack-sign=
ature/sender_notified.gif" alt=3D"Mailtrack" class=3D"" width=3D"32" height=
=3D"32">
                    </a>
                </td>
                <td>
                    <span style=3D"color:#777">Sender notified by</span> <b=
r>
                    <a href=3D"https://mailtrack.io?utm_source=3Dgmail&amp;=
utm_medium=3Dsignature&amp;utm_campaign=3Dsignaturevirality5&amp;" class=3D=
"mt-install" style=3D"color:#4374f7">Mailtrack</a>
                    <span style=3D"color:transparent;font-size:0">11/30/18,=
 5:14:43 PM</span>
                </td>
                <td>
                   =20
                </td>
            </tr>
        </tbody></table>
    </div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr">On Fri, 30 =
Nov 2018 at 15:40, Tony V E &lt;<a href=3D"mailto:tvaneerd@gmail.com" targe=
t=3D"_blank">tvaneerd@gmail.com</a>&gt; wrote:<br></div><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div lang=3D"en-US" style=3D"background-color:rgb(255,255,255=
);line-height:initial">                                                    =
                                  <div style=3D"width:100%;font-size:initia=
l;font-family:Calibri,&#39;Slate Pro&#39;,sans-serif,sans-serif;color:rgb(3=
1,73,125);text-align:initial;background-color:rgb(255,255,255)">Who decides=
 the parallelism? The stream?</div><div style=3D"width:100%;font-size:initi=
al;font-family:Calibri,&#39;Slate Pro&#39;,sans-serif,sans-serif;color:rgb(=
31,73,125);text-align:initial;background-color:rgb(255,255,255)"><br></div>=
<div style=3D"width:100%;font-size:initial;font-family:Calibri,&#39;Slate P=
ro&#39;,sans-serif,sans-serif;color:rgb(31,73,125);text-align:initial;backg=
round-color:rgb(255,255,255)">Your stream might make a good high level API,=
 but we tend to first standardize lower level APIs (from which you can buil=
d higher APIs). Like executers and queues and ranges.</div><div style=3D"wi=
dth:100%;font-size:initial;font-family:Calibri,&#39;Slate Pro&#39;,sans-ser=
if,sans-serif;color:rgb(31,73,125);text-align:initial;background-color:rgb(=
255,255,255)"><br></div><div style=3D"width:100%;font-size:initial;font-fam=
ily:Calibri,&#39;Slate Pro&#39;,sans-serif,sans-serif;color:rgb(31,73,125);=
text-align:initial;background-color:rgb(255,255,255)">(it may be a mistake =
that we focus so much on low level, but if we don&#39;t give tools that off=
er 100% performance for all cases (ie by allowing custom allocators, etc) t=
hen people just write there own anyhow.)</div><div style=3D"width:100%;font=
-size:initial;font-family:Calibri,&#39;Slate Pro&#39;,sans-serif,sans-serif=
;color:rgb(31,73,125);text-align:initial;background-color:rgb(255,255,255)"=
><br></div><div style=3D"width:100%;font-size:initial;font-family:Calibri,&=
#39;Slate Pro&#39;,sans-serif,sans-serif;color:rgb(31,73,125);text-align:in=
itial;background-color:rgb(255,255,255)">If we were to separate your APIs i=
nto parts, what do we end up with?</div><div style=3D"width:100%;font-size:=
initial;font-family:Calibri,&#39;Slate Pro&#39;,sans-serif,sans-serif;color=
:rgb(31,73,125);text-align:initial;background-color:rgb(255,255,255)"><br><=
/div>                                                                      =
                                                               <div style=
=3D"width:100%;font-size:initial;font-family:Calibri,&#39;Slate Pro&#39;,sa=
ns-serif,sans-serif;color:rgb(31,73,125);text-align:initial;background-colo=
r:rgb(255,255,255)"><br style=3D"display:initial"></div>                   =
                                                                           =
                                                                           =
                          <div style=3D"font-size:initial;font-family:Calib=
ri,&#39;Slate Pro&#39;,sans-serif,sans-serif;color:rgb(31,73,125);text-alig=
n:initial;background-color:rgb(255,255,255)">Sent=C2=A0from=C2=A0my=C2=A0Bl=
ackBerry=C2=A0portable=C2=A0Babbage=C2=A0Device</div>                      =
                                                                           =
                                                                           =
      <table width=3D"100%" style=3D"background-color:white;border-spacing:=
0px"> <tbody><tr><td colspan=3D"2" style=3D"font-size:initial;text-align:in=
itial;background-color:rgb(255,255,255)">                           <div st=
yle=3D"border-style:solid none none;border-top-color:rgb(181,196,223);borde=
r-top-width:1pt;padding:3pt 0in 0in;font-family:Tahoma,&#39;BB Alpha Sans&#=
39;,&#39;Slate Pro&#39;;font-size:10pt">  <div><b>From: </b>Vu Pham</div><d=
iv><b>Sent: </b>Friday, November 30, 2018 8:15 AM</div><div><b>To: </b><a h=
ref=3D"mailto:std-proposals@isocpp.org" target=3D"_blank">std-proposals@iso=
cpp.org</a></div><div><b>Reply To: </b><a href=3D"mailto:std-proposals@isoc=
pp.org" target=3D"_blank">std-proposals@isocpp.org</a></div><div><b>Subject=
: </b>Re: [std-proposals] Asynchronously processing elements of stream in C=
++</div></div></td></tr></tbody></table><div style=3D"border-style:solid no=
ne none;border-top-color:rgb(186,188,209);border-top-width:1pt;font-size:in=
itial;text-align:initial;background-color:rgb(255,255,255)"></div><br><div =
id=3D"m_-3292488621700084380m_5863383336927937912_originalContent"><div dir=
=3D"ltr"><div>I think I had an overdose of caffein and came up with another=
 API to make this more like C++. The idea is to use the extraction and inse=
rtion operators on streams:</div><div><span style=3D"font-family:monospace,=
monospace"><br></span></div><div><span style=3D"font-family:monospace,monos=
pace">stream&lt;T&gt; source(....)</span></div><div><span style=3D"font-fam=
ily:monospace,monospace">sink&lt;U&gt; my_sink(...)<br></span></div><div><s=
pan style=3D"font-family:monospace,monospace">function&lt;U(T&amp;)&gt; map=
_fn =3D ...=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 // a map function<br></span></div><div><span style=3D"font-family:monos=
pace,monospace">function&lt;void(U&amp;)&gt; foreach_fn =3D ...=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 // a foreach function</span></di=
v><div><span style=3D"font-family:monospace,monospace"><br></span></div><di=
v><span style=3D"font-family:monospace,monospace">// the API:</span></div><=
div><span style=3D"font-family:monospace,monospace">source &gt;&gt; mapper(=
map_fn) &gt;&gt; foreach(foreach_fn);</span></div><div><span style=3D"font-=
family:monospace,monospace"><br></span></div><div><span style=3D"font-famil=
y:monospace,monospace">// intermediate results can be stored:</span></div><=
div><span style=3D"font-family:monospace,monospace">auto mapped =3D source =
&gt;&gt; mapper(map_fn);<br></span></div><div><span style=3D"font-family:mo=
nospace,monospace">mapped &gt;&gt; write(my_sink);</span></div><div><br></d=
iv><div>- <span style=3D"font-family:monospace,monospace">mapper(), foreach=
(), write().</span>.. are utility boilerplate that turn<span style=3D"font-=
family:monospace,monospace"> std::function&lt;..&gt;</span> into an instanc=
e of <span style=3D"font-family:monospace,monospace">stream_processor</span=
>.</div><div>- the API <span style=3D"font-family:monospace,monospace">sour=
ce &gt;&gt; mapper(map_fn)</span> is implemented as an overload of <span st=
yle=3D"font-family:monospace,monospace">operator&gt;&gt;(stream&amp; s, str=
eam_processor&amp; processor)</span>, that returns another <span style=3D"f=
ont-family:monospace,monospace">stream</span> or <span style=3D"font-family=
:monospace,monospace">void</span>, depending on the type of the given <span=
 style=3D"font-family:monospace,monospace">stream_processor</span>.<br></di=
v><div><br></div><div>I think this can be a C++ way to construct complicate=
d pipelines (I almost feel like a C++ fashionista).</div><div>The extreme o=
f this is to get rid of <span style=3D"font-family:monospace,monospace">map=
per(), foreach()</span>.. altogether: <span style=3D"font-family:monospace,=
monospace">source &gt;&gt; map_fn &gt;&gt; foreach_fn;</span></div><div>How=
ever it may be confusing for developers.</div><div><br></div><div>For paral=
lel collections, the summary operations can be supported similarly:</div><d=
iv><span style=3D"font-family:monospace,monospace">future&lt;U&gt; result =
=3D source &gt;&gt; mapper(map_fn) &gt;&gt; aggregator(&lt;initial value&gt=
;, &lt;combiner&gt;, &lt;aggregator&gt;);</span></div><div><br></div><div>W=
hat do you think?</div><div>If this doesn&#39;t look good enough, I will do=
uble my caffein consumption and may come up with something better (or worse=
, no promise).</div><div><br></div><div>Cheers,</div><div>Vu<br></div><div>=
<br></div><div><br></div><div><br></div></div><br><div class=3D"gmail_quote=
"><div dir=3D"ltr">On Thu, Nov 22, 2018 at 3:45 PM Vu Pham &lt;<a href=3D"m=
ailto:phvu225@gmail.com" target=3D"_blank">phvu225@gmail.com</a>&gt; wrote:=
<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bord=
er-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>Thanks for t=
he pointers. It looks like C++ is getting weirder :)</div><div><br></div><d=
iv>Since I mentioned 2 different things, let&#39;s consider each of them an=
d why they should be part of the standard library. Sorry in advance since t=
his is going to be a monologue with lots of (potentially ill-informed) opin=
ions.<br></div><div><br></div><div>The stream&lt;T&gt; construct is a patte=
rn that can be used in a wide variety of applications, including web servic=
es, data analysis (processing logs, for example), distributed computing sys=
tems (especially for people who do scientific computing)...</div><div>Peopl=
e came up with many different implementations for asynchronous event-driven=
 networking libraries for C++, but they tend to be quite scattered.</div><d=
iv>The stream&lt;T&gt; may serve as a lingua-franca for these kinds of syst=
ems in C++.</div><div>In term of APIs, the closest examples in other langua=
ges I can find are Apache Beam, Flink, Spark. All of them are for the JVM.<=
br></div><div><br></div><div>The &quot;parallel collection&quot; construct =
is an easy way to do things in parallel without messing around with threads=
..</div><div>You may be thinking our vectors and lists are usually not that =
big, and may not be very beneficial to parallelize. <br></div><div>But I th=
ink part of the problem is it has never been so easy to parallelize algorit=
hms on collections, that&#39;s why we settle with our for-loops and be good=
 with it.<br></div><div>Now, this construct will allow us to rethink a lot =
of collection-based algorithms. For example, I will be very happy to replac=
e qsort() with a parallel, ideally non-blocking, version of it.</div><div><=
br></div><div>There are similar ideas in other languages: LINQ has a strong=
 SQL flavor and heavily extends the language, which may not really be what =
we want.</div><div>Scala has parallel versions for all of its collections, =
which is great in term of ease-of-use, but I think maintaining the consiste=
ncy between the sequential and parallel version will require extra work.</d=
iv><div>By giving users a mininal set of abstractions, we don&#39;t try to =
be too opinionated, while allowing it to be implemented efficiently on diff=
erent hardware.<br></div><div></div><div><br></div><div>It is hard for me t=
o categorize these constructs. They fall somewhere in the intersection betw=
een parallelism, functional programming, and distributed system, big data, =
etc. if I must.</div><div><br></div><div>The only concern for me is these c=
onstructs don&#39;t feel very C++ (like std::async, for example). The first=
 time I saw std::async, I thought: right, that&#39;s what a C++ dude will d=
o, not an Executor.</div><div>I understand it will take a lot of time for s=
omething to become a standard, so I am sharing these ideas so that maybe we=
 can make them more like a C++ construct.<br></div><div>Using pipes like in=
 RxCpp is a curious way to do it.</div><div><br></div><div>I come from an e=
ngineering background where I tend to build distributed systems to crunch n=
umbers.</div><div>The JVM is great for building distributed systems at ease=
, but not so great for crunching numbers.</div><div>C++ is great for crunch=
ing numbers (with BLAS and friends), but not so great for building distribu=
ted systems (you can build great distributed system in C++, but it will tak=
e a significant effort).</div><div>I think that is quite unfortunate, and w=
e can definitely expect more from C++.<br></div><div><div><br></div><div><b=
r></div><div>Btw, when I said:<br></div></div><div class=3D"gmail_quote"><b=
lockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px =
#ccc solid;padding-left:1ex"><div class=3D"gmail_quote"><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div dir=3D"ltr"><div>Unlike streams, those elements are coun=
table and stored in-memory...</div></div></blockquote></div></blockquote></=
div><div>I meant to say &quot;finite&quot; instead of &quot;countable&quot;=
, with the implication that streams are infinite, although nothing is pract=
ically infinite in our profession :)</div></div>
</blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr" class=3D"m=
_-3292488621700084380m_5863383336927937912gmail_signature" data-smartmail=
=3D"gmail_signature"><div dir=3D"ltr"><div><div dir=3D"ltr"><div>PHAM Hoai =
Vu</div><div>LinkedIn:=C2=A0<a href=3D"http://linkedin.com/in/vuphoai" targ=
et=3D"_blank">linkedin.com/in/vuphoai</a></div></div></div></div></div>

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org" target=3D"_=
blank">std-proposals+unsubscribe@isocpp.org</a>.<br>
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org" target=3D"_blank">std-proposals@isocpp.org</a>.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/CAFhKpXEjnzNnivG%2BH7dYdefqu%2BcBupy5=
db_0YduhZpiPrfwRvQ%40mail.gmail.com?utm_medium=3Demail&amp;utm_source=3Dfoo=
ter" target=3D"_blank">https://groups.google.com/a/isocpp.org/d/msgid/std-p=
roposals/CAFhKpXEjnzNnivG%2BH7dYdefqu%2BcBupy5db_0YduhZpiPrfwRvQ%40mail.gma=
il.com</a>.<br>
<br></div></div>

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org" target=3D"_=
blank">std-proposals+unsubscribe@isocpp.org</a>.<br>
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org" target=3D"_blank">std-proposals@isocpp.org</a>.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/20181130154025.5181497.36363.67131%40=
gmail.com?utm_medium=3Demail&amp;utm_source=3Dfooter" target=3D"_blank">htt=
ps://groups.google.com/a/isocpp.org/d/msgid/std-proposals/20181130154025.51=
81497.36363.67131%40gmail.com</a>.<br>
</blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"=
 class=3D"m_-3292488621700084380gmail_signature" data-smartmail=3D"gmail_si=
gnature"><div dir=3D"ltr"><div><div dir=3D"ltr"><span style=3D"font-size:12=
..8px">Regards,</span><div style=3D"font-size:12.8px"><br></div><div style=
=3D"font-size:12.8px">Gaetano Checinski</div><div style=3D"font-size:12.8px=
"><span style=3D"font-size:12.8px">Founder of Loopperfect</span><br></div><=
div style=3D"font-size:12.8px"><span style=3D"font-size:12.8px"><a href=3D"=
https://loopperfect.com" target=3D"_blank">https://loopperfect.com</a></spa=
n></div><div style=3D"font-size:12.8px"><span style=3D"font-size:12.8px"><a=
 href=3D"https://buckaroo.pm" target=3D"_blank">https://buckaroo.pm</a></sp=
an></div></div></div></div></div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org">std-proposa=
ls+unsubscribe@isocpp.org</a>.<br />
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org">std-proposals@isocpp.org</a>.<br />
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/CAN0NuG761OD5pMWemQJW2-OqhqUTf1Gq8G79=
5uOwtmdP6khWRQ%40mail.gmail.com?utm_medium=3Demail&utm_source=3Dfooter">htt=
ps://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAN0NuG761OD5pMWe=
mQJW2-OqhqUTf1Gq8G795uOwtmdP6khWRQ%40mail.gmail.com</a>.<br />

--00000000000076206e057be54a5a--

.

On 11/30/18 6:39 PM, Gaetano Checinski wrote:

> As a site note: I implemented a coroutine and pull based stream
> processing library: https://github.com/LoopPerfect/conduit
> The API you depicted is very similar to conduits.

Can you elaborate on the design decision to compose operations
into a stream, and why you have operations like orElse and forEach?

Given that you are using co-routines, why didn't you compose operations
by writing a co-routine with the normal C++ syntax like the if-else
statement?

--
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/66b790f8-b7cc-2f31-c646-726a9ecc366d%40mail1.stofanet.dk.

.

--000000000000a35f3a057bf9fa1a
Content-Type: text/plain; charset="UTF-8"

> why you have operations like orElse and forEach?

*lhs >> orElse(rhs)*

OrElse
<https://github.com/LoopPerfect/conduit/blob/master/include/conduit/or-else.hpp#L13>
Is just a special compositional operator that streams *rhs* stream only iff
*lhs* is an empty stream.
It is the same idea as behind getOrDefault(value) in other monads(eg.
maybe) applied to streams.

*lhs >> forEach(side-effect)*

First of all, forEach
<https://github.com/LoopPerfect/conduit/blob/master/include/conduit/foreach.hpp#L12>
is not a sink and does not transform the stream.
However this one is admitelly questionable from a FP perspective.
It's purpose is to inject a side effect into the processing pipeline.

For both of those operators other names might have been better.

> Can you elaborate on the design decision to compose operations into a
stream
We aimed to implement common functional operators we were using on a daily
basis for data processing in our various products.

> Given that you are using co-routines, why didn't you compose operations
by writing a co-routine with the normal C++ syntax like the if-else
statement?

I'm not sure what you mean. I'd argue I did. Every operator is implemented
as a coroutine.

Are you asking why you should use Conduits
<https://github.com/LoopPerfect/conduit/> operators as opposed to just
writing coroutines?

Well Conduit <https://github.com/LoopPerfect/conduit/> provides implements
high-level operations that naturally occur in stream processing.
In the process of  DRYing up your codebase you would end up refactoring
your code into those operators.
They are part of the functional programmers vocabulary.

> Can you elaborate on the design decision to compose operations
into a stream

We wanted to offer zero-cost compositional operators that work with the
current Coroutine TS.
RxCPP already provides a reactive push-based solution that works nicely
with open-systems.
We focused to build a solution for closed systems using a pull based model.

*Composability and Reusability*

We wanted to make composition cheap and easy.
We realized that we can maximize those metrics if you can defer the control
of the computation to the consumer.
This way the algorithm implementor can defer decisions like storage and
amount of iterations to the consumer of the sequence.
As a solution we implemented inversion of control via *lazyness.* As a
result all operators are lazy

*Zero-Cost*

As we wanted to enable users to write high-level abstraction, a runtime
overhead was not acceptable to us.
We were intrigued by the Coroutine TS and investigated what is doable in
its current state.
We were mostly successful but there are a couple cases where the optimizer
fails to remove a heap allocation.

*Simplicity*

We wanted to make sure it is easy to add new operators.
All existing solution are quite complicated as they try to handle and
optimize for all the different types of iterators.
Some of them implement Ranges TS' '.base' method.
We don't think it is required in any scenario and think it is a bad idea as
it encourages place oriented programming (which is the source of many bugs).

Thanks to Coroutine TS implementation is quite trivial.
Coroutine TS even type erases the handle enabling Conduit
<https://github.com/LoopPerfect/conduit/> to provide one type Seq<T> for
all coroutines.
As a result it compile times are very low compared to range-v3 for instance.

*Detour Open vs. Closed Systems or Push vs Pull*

*Open-Systems*

A system where datastreams are external events you have no control over.
Examples are timers and I/O like sockets or mouse moves.
The metrics in open systems you want to control are usually latency and
concurrency.
Those systems follow often a push based to enable the datasource to have
control over the consumer.

*Closed-Systems*

A system where data(generation) is fully controlled by the system.
Examples are simulations or parsers where determinism and predictable
performance matters.
Those systems follow often a pull based to enable the consumer to control
the computation.


There is a big overlap in what you can do in pull and push based systems.
However I don't believe it makes sense to use one

Many open systems consist of many parts that are closed systems.

HTTP Servers are here a good example:

Incoming requests are outside your control but once you received a request
you have full control over that chunk of data.
Parsing the requests using a pull based solution might be sensible.


[image: Mailtrack]
<https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&>
Sender
notified by
Mailtrack
<https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&>
12/01/18,
6:20:38 PM

On Sat, 1 Dec 2018 at 15:55, Bjorn Reese <breese@mail1.stofanet.dk> wrote:

> On 11/30/18 6:39 PM, Gaetano Checinski wrote:
>
> > As a site note: I implemented a coroutine and pull based stream
> > processing library: https://github.com/LoopPerfect/conduit
> > The API you depicted is very similar to conduits.
>
> Can you elaborate on the design decision to compose operations
> into a stream, and why you have operations like orElse and forEach?
>
> Given that you are using co-routines, why didn't you compose operations
> by writing a co-routine with the normal C++ syntax like the if-else
> statement?
>
> --
> You received this message because you are subscribed to the Google Groups
> "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to std-proposals+unsubscribe@isocpp.org.
> To post to this group, send email to std-proposals@isocpp.org.
> To view this discussion on the web visit
> https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/66b790f8-b7cc-2f31-c646-726a9ecc366d%40mail1.stofanet.dk
> .
>


--
Regards,

Gaetano Checinski
Founder of Loopperfect
https://loopperfect.com
https://buckaroo.pm

--
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAN0NuG6pe2AvWH5XTWvkiHpYF4a6VSBmvdm76QGw43z8f7PjQw%40mail.gmail.com.

--000000000000a35f3a057bf9fa1a
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><img width=3D"0" height=3D"0" class=3D"mailtrack-img" alt=
=3D"" style=3D"display:flex" src=3D"https://mailtrack.io/trace/mail/09ebd20=
23ceaddb074ce6f94bfed4c06759c4c4f.png?u=3D931501"><div></div>&gt; why you h=
ave operations like orElse and forEach?<div><br></div><div><b>lhs &gt;&gt; =
orElse(rhs)</b></div><div><br></div><div><a href=3D"https://github.com/Loop=
Perfect/conduit/blob/master/include/conduit/or-else.hpp#L13">OrElse</a> Is =
just a special compositional operator that streams <b>rhs</b> stream only i=
ff <b>lhs</b> is an empty stream.</div><div>It is the same idea as behind g=
etOrDefault(value) in other monads(eg. maybe) applied to streams.</div><div=
><br></div><div><b>lhs &gt;&gt; forEach(side-effect)</b><br><br>First of al=
l, <a href=3D"https://github.com/LoopPerfect/conduit/blob/master/include/co=
nduit/foreach.hpp#L12">forEach</a> is not a sink and does not transform the=
 stream.</div><div>However this one is admitelly questionable from a FP per=
spective.=C2=A0</div><div>It&#39;s purpose is to inject a side effect into =
the processing pipeline.</div><div><br></div><div>For both of those operato=
rs other names might have been better.</div><div><br></div><div>&gt; Can yo=
u elaborate on the design decision to compose operations into a stream</div=
><div>We aimed to implement common functional operators we were using on a =
daily basis for data processing in our various products.</div><div><br></di=
v><div><div>&gt; Given that you are using co-routines, why didn&#39;t you c=
ompose operations<br>by writing a co-routine with the normal C++ syntax lik=
e the if-else<br>statement?</div><div><br></div><div>I&#39;m not sure what =
you mean. I&#39;d argue I did. Every operator is implemented as a coroutine=
..</div><div><br></div><div>Are you asking why you should use <a href=3D"htt=
ps://github.com/LoopPerfect/conduit/">Conduits</a> operators as opposed to =
just writing coroutines?</div><div><br></div><div>Well <a href=3D"https://g=
ithub.com/LoopPerfect/conduit/">Conduit</a> provides implements high-level =
operations that naturally occur in stream processing.</div><div>In the proc=
ess of=C2=A0 DRYing up your codebase you would end up refactoring your code=
 into those operators.</div><div>They are part of the functional programmer=
s vocabulary.</div><div><br></div>&gt; Can you elaborate on the design deci=
sion to compose operations</div>into a stream<div><br></div><div>We wanted =
to offer zero-cost compositional operators that work with the current Corou=
tine TS.</div><div>RxCPP already provides a reactive push-based solution th=
at works nicely with open-systems.=C2=A0</div><div>We focused to build a so=
lution for closed systems using a pull based model.</div><div><br></div><di=
v><div><b>Composability and Reusability</b><br></div><div><br></div><div>We=
 wanted to make composition cheap and easy.</div></div><div>We realized tha=
t we can maximize those metrics if you can defer the control of the computa=
tion to the consumer.</div><div>This way the algorithm implementor can defe=
r decisions like storage and amount of iterations to the consumer of the se=
quence.</div><div>As a solution we implemented inversion of control via <b>=
lazyness.</b> As a result all operators are lazy=C2=A0</div><div><b><br></b=
></div><div><div><b>Zero-Cost</b></div><div><br></div><div>As we wanted to =
enable users to write high-level abstraction, a runtime overhead was not ac=
ceptable to us.</div></div><div>We were intrigued by the Coroutine TS and i=
nvestigated what is doable in its current state.</div><div>We were mostly s=
uccessful but there are a couple cases where the optimizer fails to remove =
a heap allocation.</div><div><br></div><div><b>Simplicity</b><br></div><div=
><b><br></b></div><div>We wanted to make sure it is easy to add new operato=
rs.=C2=A0</div><div>All existing solution are quite complicated as they try=
 to handle and optimize for all the different types of iterators.</div><div=
>Some of them implement Ranges TS&#39; &#39;.base&#39; method.=C2=A0</div><=
div>We don&#39;t think it is required in any scenario and think it is a bad=
 idea as it encourages place oriented programming (which is the source of m=
any bugs).</div><div><br></div><div>Thanks to Coroutine TS implementation i=
s quite trivial.</div><div>Coroutine TS even type erases the handle enablin=
g <a href=3D"https://github.com/LoopPerfect/conduit/">Conduit</a> to provid=
e one type Seq&lt;T&gt; for all coroutines.</div><div>As a result it compil=
e times are very low compared to range-v3 for instance.</div><div><br></div=
><div><b>Detour Open vs. Closed Systems or Push vs Pull</b></div><div><br><=
/div><b>Open-Systems</b><br><br>A system where datastreams are external eve=
nts you have no control over.<br>Examples are timers and I/O like sockets o=
r mouse moves.<br>The metrics in open systems you want to control are usual=
ly latency and concurrency.<br>Those systems follow often a push based to e=
nable the datasource to have control over the consumer.=C2=A0<br><br><b>Clo=
sed-Systems</b><br><br><div>A system where data(generation) is fully contro=
lled by the system.</div>Examples are simulations or parsers where determin=
ism and predictable performance matters.<br><div><div>Those systems follow =
often a pull based to enable the consumer to control the computation.</div>=
</div><br><div><br></div><div>There is a big overlap in what you can do in =
pull and push based systems.</div><div>However I don&#39;t believe it makes=
 sense to use one=C2=A0</div><blockquote style=3D"margin:0 0 0 40px;border:=
none;padding:0px"></blockquote>Many open systems consist of many parts that=
 are closed systems.<br><blockquote style=3D"margin:0 0 0 40px;border:none;=
padding:0px"></blockquote>HTTP Servers are here a good example:<br><blockqu=
ote style=3D"margin:0 0 0 40px;border:none;padding:0px"></blockquote>Incomi=
ng requests are outside your control but once you received a request you ha=
ve full control over that chunk of data.<br>Parsing the requests using a pu=
ll based solution might be sensible.<div><br></div><br><div class=3D"mt-sig=
nature">
        <table border=3D"0" cellpadding=3D"8" cellspacing=3D"0" style=3D"us=
er-select: none;">
            <tbody><tr>
                <td>
                    <a href=3D"https://mailtrack.io?utm_source=3Dgmail&amp;=
utm_medium=3Dsignature&amp;utm_campaign=3Dsignaturevirality5&amp;" class=3D=
"" style=3D"text-decoration:none">
                        <img src=3D"https://s3.amazonaws.com/mailtrack-sign=
ature/sender_notified.gif" alt=3D"Mailtrack" class=3D"" width=3D"32" height=
=3D"32">
                    </a>
                </td>
                <td>
                    <span style=3D"color:#777">Sender notified by</span> <b=
r>
                    <a href=3D"https://mailtrack.io?utm_source=3Dgmail&amp;=
utm_medium=3Dsignature&amp;utm_campaign=3Dsignaturevirality5&amp;" class=3D=
"mt-install" style=3D"color:#4374f7">Mailtrack</a>
                    <span style=3D"color:transparent;font-size:0">12/01/18,=
 6:20:38 PM</span>
                </td>
                <td>
                   =20
                </td>
            </tr>
        </tbody></table>
    </div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr">On Sat, 1 D=
ec 2018 at 15:55, Bjorn Reese &lt;<a href=3D"mailto:breese@mail1.stofanet.d=
k">breese@mail1.stofanet.dk</a>&gt; wrote:<br></div><blockquote class=3D"gm=
ail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-le=
ft:1ex">On 11/30/18 6:39 PM, Gaetano Checinski wrote:<br>
<br>
&gt; As a site note: I implemented a coroutine and pull based stream <br>
&gt; processing library: <a href=3D"https://github.com/LoopPerfect/conduit"=
 rel=3D"noreferrer" target=3D"_blank">https://github.com/LoopPerfect/condui=
t</a><br>
&gt; The API you depicted is very similar to conduits.<br>
<br>
Can you elaborate on the design decision to compose operations<br>
into a stream, and why you have operations like orElse and forEach?<br>
<br>
Given that you are using co-routines, why didn&#39;t you compose operations=
<br>
by writing a co-routine with the normal C++ syntax like the if-else<br>
statement?<br>
<br>
-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals%2Bunsubscribe@isocpp.org" target=3D=
"_blank">std-proposals+unsubscribe@isocpp.org</a>.<br>
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org" target=3D"_blank">std-proposals@isocpp.org</a>.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/66b790f8-b7cc-2f31-c646-726a9ecc366d%=
40mail1.stofanet.dk" rel=3D"noreferrer" target=3D"_blank">https://groups.go=
ogle.com/a/isocpp.org/d/msgid/std-proposals/66b790f8-b7cc-2f31-c646-726a9ec=
c366d%40mail1.stofanet.dk</a>.<br>
</blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"=
 class=3D"gmail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"l=
tr"><div><div dir=3D"ltr"><span style=3D"font-size:12.8px">Regards,</span><=
div style=3D"font-size:12.8px"><br></div><div style=3D"font-size:12.8px">Ga=
etano Checinski</div><div style=3D"font-size:12.8px"><span style=3D"font-si=
ze:12.8px">Founder of Loopperfect</span><br></div><div style=3D"font-size:1=
2.8px"><span style=3D"font-size:12.8px"><a href=3D"https://loopperfect.com"=
 target=3D"_blank">https://loopperfect.com</a></span></div><div style=3D"fo=
nt-size:12.8px"><span style=3D"font-size:12.8px"><a href=3D"https://buckaro=
o.pm" target=3D"_blank">https://buckaroo.pm</a></span></div></div></div></d=
iv></div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org">std-proposa=
ls+unsubscribe@isocpp.org</a>.<br />
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org">std-proposals@isocpp.org</a>.<br />
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/CAN0NuG6pe2AvWH5XTWvkiHpYF4a6VSBmvdm7=
6QGw43z8f7PjQw%40mail.gmail.com?utm_medium=3Demail&utm_source=3Dfooter">htt=
ps://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAN0NuG6pe2AvWH5X=
TWvkiHpYF4a6VSBmvdm76QGw43z8f7PjQw%40mail.gmail.com</a>.<br />

--000000000000a35f3a057bf9fa1a--

.

--000000000000c9b957057c46459c
Content-Type: text/plain; charset="UTF-8"

Lots of good ideas flying around.

I think all your questions boils down to 2 things (actually 1 thing):
- how to manage states in different stages of the stream processing
pipeline, and
- How to buffer (Of course the buffer itself can be seen as a state, but it
may need special care because it may grow really big).

If we have a good answer for both questions, all your examples can be
implemented:
- The example in "Push or Pull" can be implemented by a map() that keeps
track of the timestamp of the last message. That can be considered as a
state.
- The "alternative between elements of 2 streams" example is similar to how
you would implement a zip() operation, which is essentially a map() with
buffers for both streams.

Backpressure is tricky. Since we are doing things locally, we may
backpressure, but it doesn't quite follow the key principles.
I think we should try to make the system works in which the producer must
produce as fast as possible. If the consumer is slower, it is consumer's
problem.
In the distributed setting, this could be solved by just adding a new bunch
of consumers (on different machines).
Since we do it locally here, it is going to be a trade-off between memory
(how much you buffer) and processing power (how many new threads you can
spawn). When it is unclear, we can leave it to the users to configure.
Buffer may be actually not too bad. We can use memory-mapped files when it
is neccessary.

The question of state management is a good one. In general functions used
in these framework doesn't have to be stateless, they only need to be
thread-safe. You can certainly have processing stages that are stateful(ly)
thread-safe.
Usually the users (i.e. developers) take care of the states, with locks and
whatever they are comfortable with.

If we choose to have built-in support for states, we most likely can do so
by recruiting a ConcurrentHashMap, that will be checkpointed along with the
whole pipeline.
Elements in the state map must implement some API so that they can be
checkpointed. Everything is local here so it simplifies the job a lot.
The state map will then be passed into user-provided functions.




On Sat, Dec 1, 2018 at 6:21 PM Gaetano Checinski <
gaetano.checinski@gmail.com> wrote:

> > why you have operations like orElse and forEach?
>
> *lhs >> orElse(rhs)*
>
> OrElse
> <https://github.com/LoopPerfect/conduit/blob/master/include/conduit/or-else.hpp#L13>
> Is just a special compositional operator that streams *rhs* stream only
> iff *lhs* is an empty stream.
> It is the same idea as behind getOrDefault(value) in other monads(eg.
> maybe) applied to streams.
>
> *lhs >> forEach(side-effect)*
>
> First of all, forEach
> <https://github.com/LoopPerfect/conduit/blob/master/include/conduit/foreach.hpp#L12>
> is not a sink and does not transform the stream.
> However this one is admitelly questionable from a FP perspective.
> It's purpose is to inject a side effect into the processing pipeline.
>
> For both of those operators other names might have been better.
>
> > Can you elaborate on the design decision to compose operations into a
> stream
> We aimed to implement common functional operators we were using on a daily
> basis for data processing in our various products.
>
> > Given that you are using co-routines, why didn't you compose operations
> by writing a co-routine with the normal C++ syntax like the if-else
> statement?
>
> I'm not sure what you mean. I'd argue I did. Every operator is implemented
> as a coroutine.
>
> Are you asking why you should use Conduits
> <https://github.com/LoopPerfect/conduit/> operators as opposed to just
> writing coroutines?
>
> Well Conduit <https://github.com/LoopPerfect/conduit/> provides
> implements high-level operations that naturally occur in stream processing.
> In the process of  DRYing up your codebase you would end up refactoring
> your code into those operators.
> They are part of the functional programmers vocabulary.
>
> > Can you elaborate on the design decision to compose operations
> into a stream
>
> We wanted to offer zero-cost compositional operators that work with the
> current Coroutine TS.
> RxCPP already provides a reactive push-based solution that works nicely
> with open-systems.
> We focused to build a solution for closed systems using a pull based model.
>
> *Composability and Reusability*
>
> We wanted to make composition cheap and easy.
> We realized that we can maximize those metrics if you can defer the
> control of the computation to the consumer.
> This way the algorithm implementor can defer decisions like storage and
> amount of iterations to the consumer of the sequence.
> As a solution we implemented inversion of control via *lazyness.* As a
> result all operators are lazy
>
> *Zero-Cost*
>
> As we wanted to enable users to write high-level abstraction, a runtime
> overhead was not acceptable to us.
> We were intrigued by the Coroutine TS and investigated what is doable in
> its current state.
> We were mostly successful but there are a couple cases where the optimizer
> fails to remove a heap allocation.
>
> *Simplicity*
>
> We wanted to make sure it is easy to add new operators.
> All existing solution are quite complicated as they try to handle and
> optimize for all the different types of iterators.
> Some of them implement Ranges TS' '.base' method.
> We don't think it is required in any scenario and think it is a bad idea
> as it encourages place oriented programming (which is the source of many
> bugs).
>
> Thanks to Coroutine TS implementation is quite trivial.
> Coroutine TS even type erases the handle enabling Conduit
> <https://github.com/LoopPerfect/conduit/> to provide one type Seq<T> for
> all coroutines.
> As a result it compile times are very low compared to range-v3 for
> instance.
>
> *Detour Open vs. Closed Systems or Push vs Pull*
>
> *Open-Systems*
>
> A system where datastreams are external events you have no control over.
> Examples are timers and I/O like sockets or mouse moves.
> The metrics in open systems you want to control are usually latency and
> concurrency.
> Those systems follow often a push based to enable the datasource to have
> control over the consumer.
>
> *Closed-Systems*
>
> A system where data(generation) is fully controlled by the system.
> Examples are simulations or parsers where determinism and predictable
> performance matters.
> Those systems follow often a pull based to enable the consumer to control
> the computation.
>
>
> There is a big overlap in what you can do in pull and push based systems.
> However I don't believe it makes sense to use one
>
> Many open systems consist of many parts that are closed systems.
>
> HTTP Servers are here a good example:
>
> Incoming requests are outside your control but once you received a request
> you have full control over that chunk of data.
> Parsing the requests using a pull based solution might be sensible.
>
>
> [image: Mailtrack]
> <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&> Sender
> notified by
> Mailtrack
> <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&> 12/01/18,
> 6:20:38 PM
>
> On Sat, 1 Dec 2018 at 15:55, Bjorn Reese <breese@mail1.stofanet.dk> wrote:
>
>> On 11/30/18 6:39 PM, Gaetano Checinski wrote:
>>
>> > As a site note: I implemented a coroutine and pull based stream
>> > processing library: https://github.com/LoopPerfect/conduit
>> > The API you depicted is very similar to conduits.
>>
>> Can you elaborate on the design decision to compose operations
>> into a stream, and why you have operations like orElse and forEach?
>>
>> Given that you are using co-routines, why didn't you compose operations
>> by writing a co-routine with the normal C++ syntax like the if-else
>> statement?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "ISO C++ Standard - Future Proposals" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to std-proposals+unsubscribe@isocpp.org.
>> To post to this group, send email to std-proposals@isocpp.org.
>> To view this discussion on the web visit
>> https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/66b790f8-b7cc-2f31-c646-726a9ecc366d%40mail1.stofanet.dk
>> .
>>
>
>
> --
> Regards,
>
> Gaetano Checinski
> Founder of Loopperfect
> https://loopperfect.com
> https://buckaroo.pm
>
> --
> You received this message because you are subscribed to the Google Groups
> "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to std-proposals+unsubscribe@isocpp.org.
> To post to this group, send email to std-proposals@isocpp.org.
> To view this discussion on the web visit
> https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAN0NuG6pe2AvWH5XTWvkiHpYF4a6VSBmvdm76QGw43z8f7PjQw%40mail.gmail.com
> <https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAN0NuG6pe2AvWH5XTWvkiHpYF4a6VSBmvdm76QGw43z8f7PjQw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>


--
PHAM Hoai Vu
LinkedIn: linkedin.com/in/vuphoai

--
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAFhKpXGwkf4GB7wXO8Fyouk%2BrGobV1x_-3mS8-Pji5Ay7ZPRNw%40mail.gmail.com.

--000000000000c9b957057c46459c
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Lots of good ideas flying around.</div><div><br></div=
><div>I think all your questions boils down to 2 things (actually 1 thing):=
</div><div>- how to manage states in different stages of the stream process=
ing pipeline, and</div><div>- How to buffer (Of course the buffer itself ca=
n be seen as a state, but it may need special care because it may grow real=
ly big).</div><div><br></div><div>If we have a good answer for both questio=
ns, all your examples can be implemented:</div><div>- The example in &quot;=
Push or Pull&quot; can be implemented by a map() that keeps track of the ti=
mestamp of the last message. That can be considered as a state.</div><div>-=
 The &quot;alternative between elements of 2 streams&quot; example is simil=
ar to how you would implement a zip() operation, which is essentially a map=
() with buffers for both streams.</div><div><br></div><div>Backpressure is =
tricky. Since we are doing things locally, we may backpressure, but it does=
n&#39;t quite follow the key principles.<br></div><div></div><div>I think w=
e should try to make the system works in which the producer must produce as=
 fast as possible. If the consumer is slower, it is consumer&#39;s problem.=
<br></div><div>In the distributed setting, this could be solved by just add=
ing a new bunch of consumers (on different machines). <br></div><div>Since =
we do it locally here, it is going to be a trade-off between memory (how mu=
ch you buffer) and processing power (how many new threads you can spawn). W=
hen it is unclear, we can leave it to the users to configure.<br></div><div=
></div><div>Buffer may be actually not too bad. We can use memory-mapped fi=
les when it is neccessary.</div><div><br></div><div>The question of state m=
anagement is a good one. In general functions used in these framework doesn=
&#39;t have to be stateless, they only need to be thread-safe. You can cert=
ainly have processing stages that are stateful(ly) thread-safe.</div><div>U=
sually the users (i.e. developers) take care of the states, with locks and =
whatever they are comfortable with.</div><div><br></div><div>If we choose t=
o have built-in support for states, we most likely can do so by recruiting =
a ConcurrentHashMap, that will be checkpointed along with the whole pipelin=
e. <br></div><div>Elements in the state map must implement some API so that=
 they can be checkpointed. Everything is local here so it simplifies the jo=
b a lot.<br></div><div>The state map will then be passed into user-provided=
 functions.</div><div><br></div><div><br></div><div><br></div></div><br><di=
v class=3D"gmail_quote"><div dir=3D"ltr">On Sat, Dec 1, 2018 at 6:21 PM Gae=
tano Checinski &lt;<a href=3D"mailto:gaetano.checinski@gmail.com">gaetano.c=
hecinski@gmail.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote=
" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><=
div dir=3D"ltr"><img width=3D"0" height=3D"0" class=3D"m_622319845077243032=
2mailtrack-img" alt=3D"" style=3D"display:flex" src=3D"https://mailtrack.io=
/trace/mail/09ebd2023ceaddb074ce6f94bfed4c06759c4c4f.png?u=3D931501"><div><=
/div>&gt; why you have operations like orElse and forEach?<div><br></div><d=
iv><b>lhs &gt;&gt; orElse(rhs)</b></div><div><br></div><div><a href=3D"http=
s://github.com/LoopPerfect/conduit/blob/master/include/conduit/or-else.hpp#=
L13" target=3D"_blank">OrElse</a> Is just a special compositional operator =
that streams <b>rhs</b> stream only iff <b>lhs</b> is an empty stream.</div=
><div>It is the same idea as behind getOrDefault(value) in other monads(eg.=
 maybe) applied to streams.</div><div><br></div><div><b>lhs &gt;&gt; forEac=
h(side-effect)</b><br><br>First of all, <a href=3D"https://github.com/LoopP=
erfect/conduit/blob/master/include/conduit/foreach.hpp#L12" target=3D"_blan=
k">forEach</a> is not a sink and does not transform the stream.</div><div>H=
owever this one is admitelly questionable from a FP perspective.=C2=A0</div=
><div>It&#39;s purpose is to inject a side effect into the processing pipel=
ine.</div><div><br></div><div>For both of those operators other names might=
 have been better.</div><div><br></div><div>&gt; Can you elaborate on the d=
esign decision to compose operations into a stream</div><div>We aimed to im=
plement common functional operators we were using on a daily basis for data=
 processing in our various products.</div><div><br></div><div><div>&gt; Giv=
en that you are using co-routines, why didn&#39;t you compose operations<br=
>by writing a co-routine with the normal C++ syntax like the if-else<br>sta=
tement?</div><div><br></div><div>I&#39;m not sure what you mean. I&#39;d ar=
gue I did. Every operator is implemented as a coroutine.</div><div><br></di=
v><div>Are you asking why you should use <a href=3D"https://github.com/Loop=
Perfect/conduit/" target=3D"_blank">Conduits</a> operators as opposed to ju=
st writing coroutines?</div><div><br></div><div>Well <a href=3D"https://git=
hub.com/LoopPerfect/conduit/" target=3D"_blank">Conduit</a> provides implem=
ents high-level operations that naturally occur in stream processing.</div>=
<div>In the process of=C2=A0 DRYing up your codebase you would end up refac=
toring your code into those operators.</div><div>They are part of the funct=
ional programmers vocabulary.</div><div><br></div>&gt; Can you elaborate on=
 the design decision to compose operations</div>into a stream<div><br></div=
><div>We wanted to offer zero-cost compositional operators that work with t=
he current Coroutine TS.</div><div>RxCPP already provides a reactive push-b=
ased solution that works nicely with open-systems.=C2=A0</div><div>We focus=
ed to build a solution for closed systems using a pull based model.</div><d=
iv><br></div><div><div><b>Composability and Reusability</b><br></div><div><=
br></div><div>We wanted to make composition cheap and easy.</div></div><div=
>We realized that we can maximize those metrics if you can defer the contro=
l of the computation to the consumer.</div><div>This way the algorithm impl=
ementor can defer decisions like storage and amount of iterations to the co=
nsumer of the sequence.</div><div>As a solution we implemented inversion of=
 control via <b>lazyness.</b> As a result all operators are lazy=C2=A0</div=
><div><b><br></b></div><div><div><b>Zero-Cost</b></div><div><br></div><div>=
As we wanted to enable users to write high-level abstraction, a runtime ove=
rhead was not acceptable to us.</div></div><div>We were intrigued by the Co=
routine TS and investigated what is doable in its current state.</div><div>=
We were mostly successful but there are a couple cases where the optimizer =
fails to remove a heap allocation.</div><div><br></div><div><b>Simplicity</=
b><br></div><div><b><br></b></div><div>We wanted to make sure it is easy to=
 add new operators.=C2=A0</div><div>All existing solution are quite complic=
ated as they try to handle and optimize for all the different types of iter=
ators.</div><div>Some of them implement Ranges TS&#39; &#39;.base&#39; meth=
od.=C2=A0</div><div>We don&#39;t think it is required in any scenario and t=
hink it is a bad idea as it encourages place oriented programming (which is=
 the source of many bugs).</div><div><br></div><div>Thanks to Coroutine TS =
implementation is quite trivial.</div><div>Coroutine TS even type erases th=
e handle enabling <a href=3D"https://github.com/LoopPerfect/conduit/" targe=
t=3D"_blank">Conduit</a> to provide one type Seq&lt;T&gt; for all coroutine=
s.</div><div>As a result it compile times are very low compared to range-v3=
 for instance.</div><div><br></div><div><b>Detour Open vs. Closed Systems o=
r Push vs Pull</b></div><div><br></div><b>Open-Systems</b><br><br>A system =
where datastreams are external events you have no control over.<br>Examples=
 are timers and I/O like sockets or mouse moves.<br>The metrics in open sys=
tems you want to control are usually latency and concurrency.<br>Those syst=
ems follow often a push based to enable the datasource to have control over=
 the consumer.=C2=A0<br><br><b>Closed-Systems</b><br><br><div>A system wher=
e data(generation) is fully controlled by the system.</div>Examples are sim=
ulations or parsers where determinism and predictable performance matters.<=
br><div><div>Those systems follow often a pull based to enable the consumer=
 to control the computation.</div></div><br><div><br></div><div>There is a =
big overlap in what you can do in pull and push based systems.</div><div>Ho=
wever I don&#39;t believe it makes sense to use one=C2=A0</div><blockquote =
style=3D"margin:0 0 0 40px;border:none;padding:0px"></blockquote>Many open =
systems consist of many parts that are closed systems.<br><blockquote style=
=3D"margin:0 0 0 40px;border:none;padding:0px"></blockquote>HTTP Servers ar=
e here a good example:<br><blockquote style=3D"margin:0 0 0 40px;border:non=
e;padding:0px"></blockquote>Incoming requests are outside your control but =
once you received a request you have full control over that chunk of data.<=
br>Parsing the requests using a pull based solution might be sensible.<div>=
<br></div><br><div class=3D"m_6223198450772430322mt-signature">
        <table border=3D"0" cellpadding=3D"8" cellspacing=3D"0">
            <tbody><tr>
                <td>
                    <a href=3D"https://mailtrack.io?utm_source=3Dgmail&amp;=
utm_medium=3Dsignature&amp;utm_campaign=3Dsignaturevirality5&amp;" style=3D=
"text-decoration:none" target=3D"_blank">
                        <img src=3D"https://s3.amazonaws.com/mailtrack-sign=
ature/sender_notified.gif" alt=3D"Mailtrack" width=3D"32" height=3D"32">
                    </a>
                </td>
                <td>
                    <span style=3D"color:#777">Sender notified by</span> <b=
r>
                    <a href=3D"https://mailtrack.io?utm_source=3Dgmail&amp;=
utm_medium=3Dsignature&amp;utm_campaign=3Dsignaturevirality5&amp;" class=3D=
"m_6223198450772430322mt-install" style=3D"color:#4374f7" target=3D"_blank"=
>Mailtrack</a>
                    <span style=3D"color:transparent;font-size:0">12/01/18,=
 6:20:38 PM</span>
                </td>
                <td>
                   =20
                </td>
            </tr>
        </tbody></table>
    </div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr">On Sat, 1 D=
ec 2018 at 15:55, Bjorn Reese &lt;<a href=3D"mailto:breese@mail1.stofanet.d=
k" target=3D"_blank">breese@mail1.stofanet.dk</a>&gt; wrote:<br></div><bloc=
kquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #cc=
c solid;padding-left:1ex">On 11/30/18 6:39 PM, Gaetano Checinski wrote:<br>
<br>
&gt; As a site note: I implemented a coroutine and pull based stream <br>
&gt; processing library: <a href=3D"https://github.com/LoopPerfect/conduit"=
 rel=3D"noreferrer" target=3D"_blank">https://github.com/LoopPerfect/condui=
t</a><br>
&gt; The API you depicted is very similar to conduits.<br>
<br>
Can you elaborate on the design decision to compose operations<br>
into a stream, and why you have operations like orElse and forEach?<br>
<br>
Given that you are using co-routines, why didn&#39;t you compose operations=
<br>
by writing a co-routine with the normal C++ syntax like the if-else<br>
statement?<br>
<br>
-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals%2Bunsubscribe@isocpp.org" target=3D=
"_blank">std-proposals+unsubscribe@isocpp.org</a>.<br>
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org" target=3D"_blank">std-proposals@isocpp.org</a>.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/66b790f8-b7cc-2f31-c646-726a9ecc366d%=
40mail1.stofanet.dk" rel=3D"noreferrer" target=3D"_blank">https://groups.go=
ogle.com/a/isocpp.org/d/msgid/std-proposals/66b790f8-b7cc-2f31-c646-726a9ec=
c366d%40mail1.stofanet.dk</a>.<br>
</blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"=
 class=3D"m_6223198450772430322gmail_signature" data-smartmail=3D"gmail_sig=
nature"><div dir=3D"ltr"><div><div dir=3D"ltr"><span style=3D"font-size:12.=
8px">Regards,</span><div style=3D"font-size:12.8px"><br></div><div style=3D=
"font-size:12.8px">Gaetano Checinski</div><div style=3D"font-size:12.8px"><=
span style=3D"font-size:12.8px">Founder of Loopperfect</span><br></div><div=
 style=3D"font-size:12.8px"><span style=3D"font-size:12.8px"><a href=3D"htt=
ps://loopperfect.com" target=3D"_blank">https://loopperfect.com</a></span><=
/div><div style=3D"font-size:12.8px"><span style=3D"font-size:12.8px"><a hr=
ef=3D"https://buckaroo.pm" target=3D"_blank">https://buckaroo.pm</a></span>=
</div></div></div></div></div>

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org" target=3D"_=
blank">std-proposals+unsubscribe@isocpp.org</a>.<br>
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org" target=3D"_blank">std-proposals@isocpp.org</a>.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/CAN0NuG6pe2AvWH5XTWvkiHpYF4a6VSBmvdm7=
6QGw43z8f7PjQw%40mail.gmail.com?utm_medium=3Demail&amp;utm_source=3Dfooter"=
 target=3D"_blank">https://groups.google.com/a/isocpp.org/d/msgid/std-propo=
sals/CAN0NuG6pe2AvWH5XTWvkiHpYF4a6VSBmvdm76QGw43z8f7PjQw%40mail.gmail.com</=
a>.<br>
</blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr" class=3D"g=
mail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><div><d=
iv dir=3D"ltr"><div>PHAM Hoai Vu</div><div>LinkedIn:=C2=A0<a href=3D"http:/=
/linkedin.com/in/vuphoai" target=3D"_blank">linkedin.com/in/vuphoai</a></di=
v></div></div></div></div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org">std-proposa=
ls+unsubscribe@isocpp.org</a>.<br />
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org">std-proposals@isocpp.org</a>.<br />
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/CAFhKpXGwkf4GB7wXO8Fyouk%2BrGobV1x_-3=
mS8-Pji5Ay7ZPRNw%40mail.gmail.com?utm_medium=3Demail&utm_source=3Dfooter">h=
ttps://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAFhKpXGwkf4GB7=
wXO8Fyouk%2BrGobV1x_-3mS8-Pji5Ay7ZPRNw%40mail.gmail.com</a>.<br />

--000000000000c9b957057c46459c--

.

------=_Part_202_1644842083.1544076253655
Content-Type: multipart/alternative;
 boundary="----=_Part_203_2022512285.1544076253656"

------=_Part_203_2022512285.1544076253656
Content-Type: text/plain; charset="UTF-8"

Hi,

I am arriving late to this thread. I am happy to see more ideas and
discussion on this topic. This space is a passion of mine. This year I
joined Eric Niebler and Lewis Baker to build async ranges in a way that
integrates coroutines and executors with the existing ranges library.

Executors
-------------

We realized earlier this year that the first step was to be involved in the
Executors design. We wrote a paper (http://wg21.link/P1055r0) to start the
conversation. If you look at the paper you will see that the design is
planned to extend to support Future and Stream (not with those names). The
paper resulted in a meeting in Bellevue, before CppCon, to agree on a
unified approach for Executors. This was followed by a paper
(https://wg21.link/P1194r0) at the San Diego meeting describing the agreed
approach for Executors design.

Concepts
-------------

The core idea in the papers we have written is to define a set of Concepts
(rather than types) for async values similar to how Iterators define
sequences of sync values. The SanDiego post-mailing should have a paper
(http://wg21.link/P1341R0)(?) that will show the work Lewis has been doing
to combine the coroutines-ts with Senders in a way that can be applied
generally for async in C++.

Push and Pull
-------------------

The Sender/Receiver concepts can be described in different ways. They are
formally-defined callbacks. They are Async Iterators. They are Streams.
They Push values.

The Coroutines-TS enables the compiler to transform a function that looks
like it is synchronous Pull into an implementation that is Push. Another
way to express this is that coroutines-TS is sugar syntax to hide callbacks.

When Senders and Coroutines-TS are combined, the same expression can be
used in the Push model or the Pull model as desired.

Pipelines
------------

There was another paper at SanDiego to add pipelines to C++
(http://wg21.link/p1261r0). This paper came from the hetero-c group in
SG14. I am working with them to write a paper that will incorporate the
Sender/Receiver model.

Implementation
--------------------

There is an implementation under active development here
(https://github.com/facebook/folly/tree/master/folly/experimental/pushmi)
To see that implementation in action (https://godbolt.org/z/woVAi9)

RXCPP
----------

I also worked on rxcpp. rxcpp supports both operator| and operator>>. rxcpp
also supports chaining with 'dot' source().map(..).filter(..).subscribe();
The original preference was 'dot' operator| was adopted to align with the
range library which is now in the standard. pushmi and rxcpp are solving
the same problem that the range library solves, they Push values
distributed in time where range Pull values distributed in space. It helps
usage when they share the same structure and composition.

Contribute
--------------

Please do play with existing libraries and get involved with feedback,
papers, etc.. The goal is to have a solid foundation for async in C++ and
the more perspectives the better.

Kirk

On Wednesday, December 5, 2018 at 5:22:26 AM UTC-8, Vu Pham wrote:
>
> Lots of good ideas flying around.
>
> I think all your questions boils down to 2 things (actually 1 thing):
> - how to manage states in different stages of the stream processing
> pipeline, and
> - How to buffer (Of course the buffer itself can be seen as a state, but
> it may need special care because it may grow really big).
>
> If we have a good answer for both questions, all your examples can be
> implemented:
> - The example in "Push or Pull" can be implemented by a map() that keeps
> track of the timestamp of the last message. That can be considered as a
> state.
> - The "alternative between elements of 2 streams" example is similar to
> how you would implement a zip() operation, which is essentially a map()
> with buffers for both streams.
>
> Backpressure is tricky. Since we are doing things locally, we may
> backpressure, but it doesn't quite follow the key principles.
> I think we should try to make the system works in which the producer must
> produce as fast as possible. If the consumer is slower, it is consumer's
> problem.
> In the distributed setting, this could be solved by just adding a new
> bunch of consumers (on different machines).
> Since we do it locally here, it is going to be a trade-off between memory
> (how much you buffer) and processing power (how many new threads you can
> spawn). When it is unclear, we can leave it to the users to configure.
> Buffer may be actually not too bad. We can use memory-mapped files when it
> is neccessary.
>
> The question of state management is a good one. In general functions used
> in these framework doesn't have to be stateless, they only need to be
> thread-safe. You can certainly have processing stages that are stateful(ly)
> thread-safe.
> Usually the users (i.e. developers) take care of the states, with locks
> and whatever they are comfortable with.
>
> If we choose to have built-in support for states, we most likely can do so
> by recruiting a ConcurrentHashMap, that will be checkpointed along with the
> whole pipeline.
> Elements in the state map must implement some API so that they can be
> checkpointed. Everything is local here so it simplifies the job a lot.
> The state map will then be passed into user-provided functions.
>
>
>
>
> On Sat, Dec 1, 2018 at 6:21 PM Gaetano Checinski <gaetano....@gmail.com
> <javascript:>> wrote:
>
>> > why you have operations like orElse and forEach?
>>
>> *lhs >> orElse(rhs)*
>>
>> OrElse
>> <https://github.com/LoopPerfect/conduit/blob/master/include/conduit/or-else.hpp#L13>
>> Is just a special compositional operator that streams *rhs* stream only
>> iff *lhs* is an empty stream.
>> It is the same idea as behind getOrDefault(value) in other monads(eg.
>> maybe) applied to streams.
>>
>> *lhs >> forEach(side-effect)*
>>
>> First of all, forEach
>> <https://github.com/LoopPerfect/conduit/blob/master/include/conduit/foreach.hpp#L12>
>> is not a sink and does not transform the stream.
>> However this one is admitelly questionable from a FP perspective.
>> It's purpose is to inject a side effect into the processing pipeline.
>>
>> For both of those operators other names might have been better.
>>
>> > Can you elaborate on the design decision to compose operations into a
>> stream
>> We aimed to implement common functional operators we were using on a
>> daily basis for data processing in our various products.
>>
>> > Given that you are using co-routines, why didn't you compose operations
>> by writing a co-routine with the normal C++ syntax like the if-else
>> statement?
>>
>> I'm not sure what you mean. I'd argue I did. Every operator is
>> implemented as a coroutine.
>>
>> Are you asking why you should use Conduits
>> <https://github.com/LoopPerfect/conduit/> operators as opposed to just
>> writing coroutines?
>>
>> Well Conduit <https://github.com/LoopPerfect/conduit/> provides
>> implements high-level operations that naturally occur in stream processing.
>> In the process of  DRYing up your codebase you would end up refactoring
>> your code into those operators.
>> They are part of the functional programmers vocabulary.
>>
>> > Can you elaborate on the design decision to compose operations
>> into a stream
>>
>> We wanted to offer zero-cost compositional operators that work with the
>> current Coroutine TS.
>> RxCPP already provides a reactive push-based solution that works nicely
>> with open-systems.
>> We focused to build a solution for closed systems using a pull based
>> model.
>>
>> *Composability and Reusability*
>>
>> We wanted to make composition cheap and easy.
>> We realized that we can maximize those metrics if you can defer the
>> control of the computation to the consumer.
>> This way the algorithm implementor can defer decisions like storage and
>> amount of iterations to the consumer of the sequence.
>> As a solution we implemented inversion of control via *lazyness.* As a
>> result all operators are lazy
>>
>> *Zero-Cost*
>>
>> As we wanted to enable users to write high-level abstraction, a runtime
>> overhead was not acceptable to us.
>> We were intrigued by the Coroutine TS and investigated what is doable in
>> its current state.
>> We were mostly successful but there are a couple cases where the
>> optimizer fails to remove a heap allocation.
>>
>> *Simplicity*
>>
>> We wanted to make sure it is easy to add new operators.
>> All existing solution are quite complicated as they try to handle and
>> optimize for all the different types of iterators.
>> Some of them implement Ranges TS' '.base' method.
>> We don't think it is required in any scenario and think it is a bad idea
>> as it encourages place oriented programming (which is the source of many
>> bugs).
>>
>> Thanks to Coroutine TS implementation is quite trivial.
>> Coroutine TS even type erases the handle enabling Conduit
>> <https://github.com/LoopPerfect/conduit/> to provide one type Seq<T> for
>> all coroutines.
>> As a result it compile times are very low compared to range-v3 for
>> instance.
>>
>> *Detour Open vs. Closed Systems or Push vs Pull*
>>
>> *Open-Systems*
>>
>> A system where datastreams are external events you have no control over.
>> Examples are timers and I/O like sockets or mouse moves.
>> The metrics in open systems you want to control are usually latency and
>> concurrency.
>> Those systems follow often a push based to enable the datasource to have
>> control over the consumer.
>>
>> *Closed-Systems*
>>
>> A system where data(generation) is fully controlled by the system.
>> Examples are simulations or parsers where determinism and predictable
>> performance matters.
>> Those systems follow often a pull based to enable the consumer to control
>> the computation.
>>
>>
>> There is a big overlap in what you can do in pull and push based systems.
>> However I don't believe it makes sense to use one
>>
>> Many open systems consist of many parts that are closed systems.
>>
>> HTTP Servers are here a good example:
>>
>> Incoming requests are outside your control but once you received a
>> request you have full control over that chunk of data.
>> Parsing the requests using a pull based solution might be sensible.
>>
>>
>> [image: Mailtrack]
>> <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&> Sender
>> notified by
>> Mailtrack
>> <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&> 12/01/18,
>> 6:20:38 PM
>>
>> On Sat, 1 Dec 2018 at 15:55, Bjorn Reese <bre...@mail1.stofanet.dk
>> <javascript:>> wrote:
>>
>>> On 11/30/18 6:39 PM, Gaetano Checinski wrote:
>>>
>>> > As a site note: I implemented a coroutine and pull based stream
>>> > processing library: https://github.com/LoopPerfect/conduit
>>> > The API you depicted is very similar to conduits.
>>>
>>> Can you elaborate on the design decision to compose operations
>>> into a stream, and why you have operations like orElse and forEach?
>>>
>>> Given that you are using co-routines, why didn't you compose operations
>>> by writing a co-routine with the normal C++ syntax like the if-else
>>> statement?
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "ISO C++ Standard - Future Proposals" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to std-proposal...@isocpp.org <javascript:>.
>>> To post to this group, send email to std-pr...@isocpp.org <javascript:>.
>>> To view this discussion on the web visit
>>> https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/66b790f8-b7cc-2f31-c646-726a9ecc366d%40mail1.stofanet.dk
>>> .
>>>
>>
>>
>> --
>> Regards,
>>
>> Gaetano Checinski
>> Founder of Loopperfect
>> https://loopperfect.com
>> https://buckaroo.pm
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "ISO C++ Standard - Future Proposals" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to std-proposal...@isocpp.org <javascript:>.
>> To post to this group, send email to std-pr...@isocpp.org <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAN0NuG6pe2AvWH5XTWvkiHpYF4a6VSBmvdm76QGw43z8f7PjQw%40mail.gmail.com
>> <https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAN0NuG6pe2AvWH5XTWvkiHpYF4a6VSBmvdm76QGw43z8f7PjQw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>
>
> --
> PHAM Hoai Vu
> LinkedIn: linkedin.com/in/vuphoai
>

--
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/2aad8a9d-33fc-465f-af1c-b18d4b68f185%40isocpp.org.

------=_Part_203_2022512285.1544076253656
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>I am arriving late to this thread. =
I am happy to see more ideas and discussion on this topic. This space is a =
passion of mine. This year I joined Eric Niebler and Lewis Baker to build a=
sync ranges in a way that integrates coroutines and executors with the exis=
ting ranges library.=C2=A0</div><div><br></div><div>Executors</div><div>---=
----------</div><div><br></div><div>We realized earlier this year that the =
first step was to be involved in the Executors design. We wrote a paper (ht=
tp://wg21.link/P1055r0) to start the conversation. If you look at the paper=
 you will see that the design is planned to extend to support Future and St=
ream (not with those names). The paper resulted in a meeting in Bellevue, b=
efore CppCon, to agree on a unified approach for Executors. This was follow=
ed by a paper (https://wg21.link/P1194r0) at the San Diego meeting describi=
ng the agreed approach for Executors design.</div><div><br></div><div>Conce=
pts</div><div>-------------</div><div><br></div><div>The core idea in the p=
apers we have written is to define a set of Concepts (rather than types) fo=
r async values similar to how Iterators define sequences of sync values. Th=
e SanDiego post-mailing should have a paper (http://wg21.link/P1341R0)(?) t=
hat will show the work Lewis has been doing to combine the coroutines-ts wi=
th Senders in a way that can be applied generally for async in C++.</div><d=
iv><br></div><div>Push and Pull</div><div>-------------------</div><div><br=
></div><div>The Sender/Receiver concepts can be described in different ways=
.. They are formally-defined callbacks.=C2=A0They are Async Iterators. They =
are Streams. They Push values.</div><div><br></div><div>The Coroutines-TS e=
nables the compiler to transform a function that looks like it is synchrono=
us Pull into an implementation that is Push. Another way to express this is=
 that coroutines-TS is sugar syntax to hide callbacks.</div><div><br></div>=
<div>When Senders and Coroutines-TS are combined, the same expression can b=
e used in the Push model or the Pull model as desired.</div><div><br></div>=
<div>Pipelines</div><div>------------</div><div><br></div><div>There was an=
other paper at SanDiego to add pipelines to C++ (http://wg21.link/p1261r0).=
 This paper came from the hetero-c group in SG14. I am working with them to=
 write a paper that will incorporate the Sender/Receiver model.<br></div><d=
iv><br></div><div>Implementation</div><div>--------------------</div><div><=
br></div><div>There is an implementation under active development here (htt=
ps://github.com/facebook/folly/tree/master/folly/experimental/pushmi)</div>=
<div>To see that implementation in action (https://godbolt.org/z/woVAi9)</d=
iv><div><br>RXCPP</div><div>----------</div><div><br></div><div>I also work=
ed on rxcpp. rxcpp supports both operator| and operator&gt;&gt;. rxcpp also=
 supports chaining with &#39;dot&#39; source().map(..).filter(..).subscribe=
(); The original preference was &#39;dot&#39; operator| was adopted to alig=
n with the range library which is now in the standard. pushmi and rxcpp are=
 solving the same problem that the range library solves, they Push values d=
istributed in time where range Pull values distributed in space. It helps u=
sage when they share the same structure and composition.</div><div><br></di=
v><div>Contribute</div><div>--------------</div><div><br></div><div>Please =
do play with existing libraries and get involved with feedback, papers, etc=
... The goal is to have a solid foundation for async in C++ and the more per=
spectives the better.</div><div><br></div><div>Kirk<br><br>On Wednesday, De=
cember 5, 2018 at 5:22:26 AM UTC-8, Vu Pham wrote:<blockquote class=3D"gmai=
l_quote" style=3D"margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;=
padding-left: 1ex;"><div dir=3D"ltr"><div>Lots of good ideas flying around.=
</div><div><br></div><div>I think all your questions boils down to 2 things=
 (actually 1 thing):</div><div>- how to manage states in different stages o=
f the stream processing pipeline, and</div><div>- How to buffer (Of course =
the buffer itself can be seen as a state, but it may need special care beca=
use it may grow really big).</div><div><br></div><div>If we have a good ans=
wer for both questions, all your examples can be implemented:</div><div>- T=
he example in &quot;Push or Pull&quot; can be implemented by a map() that k=
eeps track of the timestamp of the last message. That can be considered as =
a state.</div><div>- The &quot;alternative between elements of 2 streams&qu=
ot; example is similar to how you would implement a zip() operation, which =
is essentially a map() with buffers for both streams.</div><div><br></div><=
div>Backpressure is tricky. Since we are doing things locally, we may backp=
ressure, but it doesn&#39;t quite follow the key principles.<br></div><div>=
</div><div>I think we should try to make the system works in which the prod=
ucer must produce as fast as possible. If the consumer is slower, it is con=
sumer&#39;s problem.<br></div><div>In the distributed setting, this could b=
e solved by just adding a new bunch of consumers (on different machines). <=
br></div><div>Since we do it locally here, it is going to be a trade-off be=
tween memory (how much you buffer) and processing power (how many new threa=
ds you can spawn). When it is unclear, we can leave it to the users to conf=
igure.<br></div><div></div><div>Buffer may be actually not too bad. We can =
use memory-mapped files when it is neccessary.</div><div><br></div><div>The=
 question of state management is a good one. In general functions used in t=
hese framework doesn&#39;t have to be stateless, they only need to be threa=
d-safe. You can certainly have processing stages that are stateful(ly) thre=
ad-safe.</div><div>Usually the users (i.e. developers) take care of the sta=
tes, with locks and whatever they are comfortable with.</div><div><br></div=
><div>If we choose to have built-in support for states, we most likely can =
do so by recruiting a ConcurrentHashMap, that will be checkpointed along wi=
th the whole pipeline. <br></div><div>Elements in the state map must implem=
ent some API so that they can be checkpointed. Everything is local here so =
it simplifies the job a lot.<br></div><div>The state map will then be passe=
d into user-provided functions.</div><div><br></div><div><br></div><div><br=
></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr">On Sat, Dec 1,=
 2018 at 6:21 PM Gaetano Checinski &lt;<a href=3D"javascript:" target=3D"_b=
lank" gdf-obfuscated-mailto=3D"ua-FtA3CAAAJ" rel=3D"nofollow" onmousedown=
=3D"this.href=3D&#39;javascript:&#39;;return true;" onclick=3D"this.href=3D=
&#39;javascript:&#39;;return true;">gaetano....@gmail.com</a>&gt; wrote:<br=
></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-=
left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><img width=3D"0" hei=
ght=3D"0" alt=3D"" style=3D"display:flex" src=3D"https://lh5.googleusercont=
ent.com/proxy/rOltQkfyYhJqivMIonhJspCN3UqokDPHAGbRgxTUaVJbqbmd3bpLvRW9jotCj=
H9ETZNg_03LRjtmUsQUdfKoVLsT4L23xGVSwh8x8mOKVsJTGHsA3KQPDDhWhdXwaXKSbTkrq3Ii=
qOI=3Dw5000-h5000"><div></div>&gt; why you have operations like orElse and =
forEach?<div><br></div><div><b>lhs &gt;&gt; orElse(rhs)</b></div><div><br><=
/div><div><a href=3D"https://github.com/LoopPerfect/conduit/blob/master/inc=
lude/conduit/or-else.hpp#L13" target=3D"_blank" rel=3D"nofollow" onmousedow=
n=3D"this.href=3D&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.=
com%2FLoopPerfect%2Fconduit%2Fblob%2Fmaster%2Finclude%2Fconduit%2For-else.h=
pp%23L13\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHGwouHatocH0nX-bnmGYg9LOyx=
5A&#39;;return true;" onclick=3D"this.href=3D&#39;https://www.google.com/ur=
l?q\x3dhttps%3A%2F%2Fgithub.com%2FLoopPerfect%2Fconduit%2Fblob%2Fmaster%2Fi=
nclude%2Fconduit%2For-else.hpp%23L13\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQj=
CNHGwouHatocH0nX-bnmGYg9LOyx5A&#39;;return true;">OrElse</a> Is just a spec=
ial compositional operator that streams <b>rhs</b> stream only iff <b>lhs</=
b> is an empty stream.</div><div>It is the same idea as behind getOrDefault=
(value) in other monads(eg. maybe) applied to streams.</div><div><br></div>=
<div><b>lhs &gt;&gt; forEach(side-effect)</b><br><br>First of all, <a href=
=3D"https://github.com/LoopPerfect/conduit/blob/master/include/conduit/fore=
ach.hpp#L12" target=3D"_blank" rel=3D"nofollow" onmousedown=3D"this.href=3D=
&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FLoopPerfect=
%2Fconduit%2Fblob%2Fmaster%2Finclude%2Fconduit%2Fforeach.hpp%23L12\x26sa\x3=
dD\x26sntz\x3d1\x26usg\x3dAFQjCNHE0UVzUS6sS8p5J_RYyN4aOWXPuA&#39;;return tr=
ue;" onclick=3D"this.href=3D&#39;https://www.google.com/url?q\x3dhttps%3A%2=
F%2Fgithub.com%2FLoopPerfect%2Fconduit%2Fblob%2Fmaster%2Finclude%2Fconduit%=
2Fforeach.hpp%23L12\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHE0UVzUS6sS8p5J=
_RYyN4aOWXPuA&#39;;return true;">forEach</a> is not a sink and does not tra=
nsform the stream.</div><div>However this one is admitelly questionable fro=
m a FP perspective.=C2=A0</div><div>It&#39;s purpose is to inject a side ef=
fect into the processing pipeline.</div><div><br></div><div>For both of tho=
se operators other names might have been better.</div><div><br></div><div>&=
gt; Can you elaborate on the design decision to compose operations into a s=
tream</div><div>We aimed to implement common functional operators we were u=
sing on a daily basis for data processing in our various products.</div><di=
v><br></div><div><div>&gt; Given that you are using co-routines, why didn&#=
39;t you compose operations<br>by writing a co-routine with the normal C++ =
syntax like the if-else<br>statement?</div><div><br></div><div>I&#39;m not =
sure what you mean. I&#39;d argue I did. Every operator is implemented as a=
 coroutine.</div><div><br></div><div>Are you asking why you should use <a h=
ref=3D"https://github.com/LoopPerfect/conduit/" target=3D"_blank" rel=3D"no=
follow" onmousedown=3D"this.href=3D&#39;https://www.google.com/url?q\x3dhtt=
ps%3A%2F%2Fgithub.com%2FLoopPerfect%2Fconduit%2F\x26sa\x3dD\x26sntz\x3d1\x2=
6usg\x3dAFQjCNEccrN_bXmHS-ziLWF3wFLZ3zqp0g&#39;;return true;" onclick=3D"th=
is.href=3D&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FL=
oopPerfect%2Fconduit%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEccrN_bXmHS=
-ziLWF3wFLZ3zqp0g&#39;;return true;">Conduits</a> operators as opposed to j=
ust writing coroutines?</div><div><br></div><div>Well <a href=3D"https://gi=
thub.com/LoopPerfect/conduit/" target=3D"_blank" rel=3D"nofollow" onmousedo=
wn=3D"this.href=3D&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub=
..com%2FLoopPerfect%2Fconduit%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcc=
rN_bXmHS-ziLWF3wFLZ3zqp0g&#39;;return true;" onclick=3D"this.href=3D&#39;ht=
tps://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FLoopPerfect%2Fcond=
uit%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEccrN_bXmHS-ziLWF3wFLZ3zqp0g=
&#39;;return true;">Conduit</a> provides implements high-level operations t=
hat naturally occur in stream processing.</div><div>In the process of=C2=A0=
 DRYing up your codebase you would end up refactoring your code into those =
operators.</div><div>They are part of the functional programmers vocabulary=
..</div><div><br></div>&gt; Can you elaborate on the design decision to comp=
ose operations</div>into a stream<div><br></div><div>We wanted to offer zer=
o-cost compositional operators that work with the current Coroutine TS.</di=
v><div>RxCPP already provides a reactive push-based solution that works nic=
ely with open-systems.=C2=A0</div><div>We focused to build a solution for c=
losed systems using a pull based model.</div><div><br></div><div><div><b>Co=
mposability and Reusability</b><br></div><div><br></div><div>We wanted to m=
ake composition cheap and easy.</div></div><div>We realized that we can max=
imize those metrics if you can defer the control of the computation to the =
consumer.</div><div>This way the algorithm implementor can defer decisions =
like storage and amount of iterations to the consumer of the sequence.</div=
><div>As a solution we implemented inversion of control via <b>lazyness.</b=
> As a result all operators are lazy=C2=A0</div><div><b><br></b></div><div>=
<div><b>Zero-Cost</b></div><div><br></div><div>As we wanted to enable users=
 to write high-level abstraction, a runtime overhead was not acceptable to =
us.</div></div><div>We were intrigued by the Coroutine TS and investigated =
what is doable in its current state.</div><div>We were mostly successful bu=
t there are a couple cases where the optimizer fails to remove a heap alloc=
ation.</div><div><br></div><div><b>Simplicity</b><br></div><div><b><br></b>=
</div><div>We wanted to make sure it is easy to add new operators.=C2=A0</d=
iv><div>All existing solution are quite complicated as they try to handle a=
nd optimize for all the different types of iterators.</div><div>Some of the=
m implement Ranges TS&#39; &#39;.base&#39; method.=C2=A0</div><div>We don&#=
39;t think it is required in any scenario and think it is a bad idea as it =
encourages place oriented programming (which is the source of many bugs).</=
div><div><br></div><div>Thanks to Coroutine TS implementation is quite triv=
ial.</div><div>Coroutine TS even type erases the handle enabling <a href=3D=
"https://github.com/LoopPerfect/conduit/" target=3D"_blank" rel=3D"nofollow=
" onmousedown=3D"this.href=3D&#39;https://www.google.com/url?q\x3dhttps%3A%=
2F%2Fgithub.com%2FLoopPerfect%2Fconduit%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x=
3dAFQjCNEccrN_bXmHS-ziLWF3wFLZ3zqp0g&#39;;return true;" onclick=3D"this.hre=
f=3D&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FLoopPer=
fect%2Fconduit%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEccrN_bXmHS-ziLWF=
3wFLZ3zqp0g&#39;;return true;">Conduit</a> to provide one type Seq&lt;T&gt;=
 for all coroutines.</div><div>As a result it compile times are very low co=
mpared to range-v3 for instance.</div><div><br></div><div><b>Detour Open vs=
.. Closed Systems or Push vs Pull</b></div><div><br></div><b>Open-Systems</b=
><br><br>A system where datastreams are external events you have no control=
 over.<br>Examples are timers and I/O like sockets or mouse moves.<br>The m=
etrics in open systems you want to control are usually latency and concurre=
ncy.<br>Those systems follow often a push based to enable the datasource to=
 have control over the consumer.=C2=A0<br><br><b>Closed-Systems</b><br><br>=
<div>A system where data(generation) is fully controlled by the system.</di=
v>Examples are simulations or parsers where determinism and predictable per=
formance matters.<br><div><div>Those systems follow often a pull based to e=
nable the consumer to control the computation.</div></div><br><div><br></di=
v><div>There is a big overlap in what you can do in pull and push based sys=
tems.</div><div>However I don&#39;t believe it makes sense to use one=C2=A0=
</div><blockquote style=3D"margin:0 0 0 40px;border:none;padding:0px"></blo=
ckquote>Many open systems consist of many parts that are closed systems.<br=
><blockquote style=3D"margin:0 0 0 40px;border:none;padding:0px"></blockquo=
te>HTTP Servers are here a good example:<br><blockquote style=3D"margin:0 0=
 0 40px;border:none;padding:0px"></blockquote>Incoming requests are outside=
 your control but once you received a request you have full control over th=
at chunk of data.<br>Parsing the requests using a pull based solution might=
 be sensible.<div><br></div><br><div>
        <table border=3D"0" cellpadding=3D"8" cellspacing=3D"0">
            <tbody><tr>
                <td>
                    <a href=3D"https://mailtrack.io?utm_source=3Dgmail&amp;=
utm_medium=3Dsignature&amp;utm_campaign=3Dsignaturevirality5&amp;" style=3D=
"text-decoration:none" target=3D"_blank" rel=3D"nofollow" onmousedown=3D"th=
is.href=3D&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fmailtrack.io%3=
Futm_source%3Dgmail%26utm_medium%3Dsignature%26utm_campaign%3Dsignaturevira=
lity5%26\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF8A2FF37ADcdnNfKW_eku-XtS2=
1g&#39;;return true;" onclick=3D"this.href=3D&#39;https://www.google.com/ur=
l?q\x3dhttps%3A%2F%2Fmailtrack.io%3Futm_source%3Dgmail%26utm_medium%3Dsigna=
ture%26utm_campaign%3Dsignaturevirality5%26\x26sa\x3dD\x26sntz\x3d1\x26usg\=
x3dAFQjCNF8A2FF37ADcdnNfKW_eku-XtS21g&#39;;return true;">
                        <img src=3D"https://lh3.googleusercontent.com/proxy=
/q9ZNY_lUioUjWnAf2wYwWWIEQ-y2RpYmdeNgRcZUWpHzY6xysu_EiwzHBd3EIWcjkgVS7aeRqd=
zO62SxcoqZHhu6ypHmdGZ-0Mu_x-EOj767ZuU=3Dw5000-h5000" alt=3D"Mailtrack" widt=
h=3D"32" height=3D"32">
                    </a>
                </td>
                <td>
                    <span style=3D"color:#777">Sender notified by</span> <b=
r>
                    <a href=3D"https://mailtrack.io?utm_source=3Dgmail&amp;=
utm_medium=3Dsignature&amp;utm_campaign=3Dsignaturevirality5&amp;" style=3D=
"color:#4374f7" target=3D"_blank" rel=3D"nofollow" onmousedown=3D"this.href=
=3D&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fmailtrack.io%3Futm_so=
urce%3Dgmail%26utm_medium%3Dsignature%26utm_campaign%3Dsignaturevirality5%2=
6\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF8A2FF37ADcdnNfKW_eku-XtS21g&#39;=
;return true;" onclick=3D"this.href=3D&#39;https://www.google.com/url?q\x3d=
https%3A%2F%2Fmailtrack.io%3Futm_source%3Dgmail%26utm_medium%3Dsignature%26=
utm_campaign%3Dsignaturevirality5%26\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQj=
CNF8A2FF37ADcdnNfKW_eku-XtS21g&#39;;return true;">Mailtrack</a>
                    <span style=3D"color:transparent;font-size:0">12/01/18,=
 6:20:38 PM</span>
                </td>
                <td>
                   =20
                </td>
            </tr>
        </tbody></table>
    </div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr">On Sat, 1 D=
ec 2018 at 15:55, Bjorn Reese &lt;<a href=3D"javascript:" target=3D"_blank"=
 gdf-obfuscated-mailto=3D"ua-FtA3CAAAJ" rel=3D"nofollow" onmousedown=3D"thi=
s.href=3D&#39;javascript:&#39;;return true;" onclick=3D"this.href=3D&#39;ja=
vascript:&#39;;return true;">bre...@mail1.stofanet.dk</a>&gt; wrote:<br></d=
iv><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left=
:1px #ccc solid;padding-left:1ex">On 11/30/18 6:39 PM, Gaetano Checinski wr=
ote:<br>
<br>
&gt; As a site note: I implemented a coroutine and pull based stream <br>
&gt; processing library: <a href=3D"https://github.com/LoopPerfect/conduit"=
 rel=3D"nofollow" target=3D"_blank" onmousedown=3D"this.href=3D&#39;https:/=
/www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FLoopPerfect%2Fconduit\x=
26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEwIt18Q5jsnY3nLny5j1VZoJKRbQ&#39;;re=
turn true;" onclick=3D"this.href=3D&#39;https://www.google.com/url?q\x3dhtt=
ps%3A%2F%2Fgithub.com%2FLoopPerfect%2Fconduit\x26sa\x3dD\x26sntz\x3d1\x26us=
g\x3dAFQjCNEwIt18Q5jsnY3nLny5j1VZoJKRbQ&#39;;return true;">https://github.c=
om/<wbr>LoopPerfect/conduit</a><br>
&gt; The API you depicted is very similar to conduits.<br>
<br>
Can you elaborate on the design decision to compose operations<br>
into a stream, and why you have operations like orElse and forEach?<br>
<br>
Given that you are using co-routines, why didn&#39;t you compose operations=
<br>
by writing a co-routine with the normal C++ syntax like the if-else<br>
statement?<br>
<br>
-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"javascript:" target=3D"_blank" gdf-obfuscated-mailto=3D"=
ua-FtA3CAAAJ" rel=3D"nofollow" onmousedown=3D"this.href=3D&#39;javascript:&=
#39;;return true;" onclick=3D"this.href=3D&#39;javascript:&#39;;return true=
;">std-proposal...@<wbr>isocpp.org</a>.<br>
To post to this group, send email to <a href=3D"javascript:" target=3D"_bla=
nk" gdf-obfuscated-mailto=3D"ua-FtA3CAAAJ" rel=3D"nofollow" onmousedown=3D"=
this.href=3D&#39;javascript:&#39;;return true;" onclick=3D"this.href=3D&#39=
;javascript:&#39;;return true;">std-pr...@isocpp.org</a>.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/66b790f8-b7cc-2f31-c646-726a9ecc366d%=
40mail1.stofanet.dk" rel=3D"nofollow" target=3D"_blank" onmousedown=3D"this=
..href=3D&#39;https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/6=
6b790f8-b7cc-2f31-c646-726a9ecc366d%40mail1.stofanet.dk&#39;;return true;" =
onclick=3D"this.href=3D&#39;https://groups.google.com/a/isocpp.org/d/msgid/=
std-proposals/66b790f8-b7cc-2f31-c646-726a9ecc366d%40mail1.stofanet.dk&#39;=
;return true;">https://groups.google.com/a/<wbr>isocpp.org/d/msgid/std-<wbr=
>proposals/66b790f8-b7cc-2f31-<wbr>c646-726a9ecc366d%40mail1.<wbr>stofanet.=
dk</a>.<br>
</blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"=
><div dir=3D"ltr"><div><div dir=3D"ltr"><span style=3D"font-size:12.8px">Re=
gards,</span><div style=3D"font-size:12.8px"><br></div><div style=3D"font-s=
ize:12.8px">Gaetano Checinski</div><div style=3D"font-size:12.8px"><span st=
yle=3D"font-size:12.8px">Founder of Loopperfect</span><br></div><div style=
=3D"font-size:12.8px"><span style=3D"font-size:12.8px"><a href=3D"https://l=
oopperfect.com" target=3D"_blank" rel=3D"nofollow" onmousedown=3D"this.href=
=3D&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Floopperfect.com\x26sa=
\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFVrT-yd5bFcviVdcWgiBypdn2Lrw&#39;;return=
 true;" onclick=3D"this.href=3D&#39;https://www.google.com/url?q\x3dhttps%3=
A%2F%2Floopperfect.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFVrT-yd5bFcv=
iVdcWgiBypdn2Lrw&#39;;return true;">https://loopperfect.com</a></span></div=
><div style=3D"font-size:12.8px"><span style=3D"font-size:12.8px"><a href=
=3D"https://buckaroo.pm" target=3D"_blank" rel=3D"nofollow" onmousedown=3D"=
this.href=3D&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuckaroo.pm\=
x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH3fWhVfx3qj3Ps1TZl1tYQfuAtdg&#39;;r=
eturn true;" onclick=3D"this.href=3D&#39;https://www.google.com/url?q\x3dht=
tps%3A%2F%2Fbuckaroo.pm\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH3fWhVfx3qj=
3Ps1TZl1tYQfuAtdg&#39;;return true;">https://buckaroo.pm</a></span></div></=
div></div></div></div>

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"javascript:" target=3D"_blank" gdf-obfuscated-mailto=3D"=
ua-FtA3CAAAJ" rel=3D"nofollow" onmousedown=3D"this.href=3D&#39;javascript:&=
#39;;return true;" onclick=3D"this.href=3D&#39;javascript:&#39;;return true=
;">std-proposal...@<wbr>isocpp.org</a>.<br>
To post to this group, send email to <a href=3D"javascript:" target=3D"_bla=
nk" gdf-obfuscated-mailto=3D"ua-FtA3CAAAJ" rel=3D"nofollow" onmousedown=3D"=
this.href=3D&#39;javascript:&#39;;return true;" onclick=3D"this.href=3D&#39=
;javascript:&#39;;return true;">std-pr...@isocpp.org</a>.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/CAN0NuG6pe2AvWH5XTWvkiHpYF4a6VSBmvdm7=
6QGw43z8f7PjQw%40mail.gmail.com?utm_medium=3Demail&amp;utm_source=3Dfooter"=
 target=3D"_blank" rel=3D"nofollow" onmousedown=3D"this.href=3D&#39;https:/=
/groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAN0NuG6pe2AvWH5XTWvk=
iHpYF4a6VSBmvdm76QGw43z8f7PjQw%40mail.gmail.com?utm_medium\x3demail\x26utm_=
source\x3dfooter&#39;;return true;" onclick=3D"this.href=3D&#39;https://gro=
ups.google.com/a/isocpp.org/d/msgid/std-proposals/CAN0NuG6pe2AvWH5XTWvkiHpY=
F4a6VSBmvdm76QGw43z8f7PjQw%40mail.gmail.com?utm_medium\x3demail\x26utm_sour=
ce\x3dfooter&#39;;return true;">https://groups.google.com/a/<wbr>isocpp.org=
/d/msgid/std-<wbr>proposals/<wbr>CAN0NuG6pe2AvWH5XTWvkiHpYF4a6V<wbr>SBmvdm7=
6QGw43z8f7PjQw%40mail.<wbr>gmail.com</a>.<br>
</blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr"><div dir=
=3D"ltr"><div><div dir=3D"ltr"><div>PHAM Hoai Vu</div><div>LinkedIn:=C2=A0<=
a href=3D"http://linkedin.com/in/vuphoai" target=3D"_blank" rel=3D"nofollow=
" onmousedown=3D"this.href=3D&#39;http://www.google.com/url?q\x3dhttp%3A%2F=
%2Flinkedin.com%2Fin%2Fvuphoai\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHSdl=
Ugs98Mc5Rut1nZPrt7LLNyAw&#39;;return true;" onclick=3D"this.href=3D&#39;htt=
p://www.google.com/url?q\x3dhttp%3A%2F%2Flinkedin.com%2Fin%2Fvuphoai\x26sa\=
x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHSdlUgs98Mc5Rut1nZPrt7LLNyAw&#39;;return =
true;">linkedin.com/in/<wbr>vuphoai</a></div></div></div></div></div>
</blockquote></div></div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;ISO C++ Standard - Future Proposals&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:std-proposals+unsubscribe@isocpp.org">std-proposa=
ls+unsubscribe@isocpp.org</a>.<br />
To post to this group, send email to <a href=3D"mailto:std-proposals@isocpp=
..org">std-proposals@isocpp.org</a>.<br />
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/isocpp.org/d/msgid/std-proposals/2aad8a9d-33fc-465f-af1c-b18d4b68f185%=
40isocpp.org?utm_medium=3Demail&utm_source=3Dfooter">https://groups.google.=
com/a/isocpp.org/d/msgid/std-proposals/2aad8a9d-33fc-465f-af1c-b18d4b68f185=
%40isocpp.org</a>.<br />

------=_Part_203_2022512285.1544076253656--

------=_Part_202_1644842083.1544076253655--

.