Rocksolid Light

Welcome to RetroBBS

mail  files  register  newsreader  groups  login

Message-ID:  

For every complex problem, there is a solution that is simple, neat, and wrong. -- H. L. Mencken

devel / comp.lang.fortran / CPU time for transcendental functions

SubjectAuthor
* CPU time for transcendental functionsRobinn
+* Re: CPU time for transcendental functionsSteven G. Kargl
|`- Re: CPU time for transcendental functionsGiorgio Pastore
`- Re: CPU time for transcendental functionsThomas Jahns

1
CPU time for transcendental functions

<657bb327$0$719$14726298@news.sunsite.dk>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=3782&group=comp.lang.fortran#3782

  copy link   Newsgroups: comp.lang.fortran
Path: i2pn2.org!i2pn.org!news.1d4.us!usenet.goja.nl.eu.org!dotsrc.org!filter.dotsrc.org!news.dotsrc.org!not-for-mail
Newsgroups: comp.lang.fortran
X-Mozilla-News-Host: news://news.dotsrc.org:119
From: 32s116e@gmail.invalid (Robinn)
Subject: CPU time for transcendental functions
Date: Fri, 15 Dec 2023 09:59:56 +0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
Firefox/91.0 SeaMonkey/2.53.18
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Lines: 28
Message-ID: <657bb327$0$719$14726298@news.sunsite.dk>
Organization: SunSITE.dk - Supporting Open source
NNTP-Posting-Host: b07f518f.news.sunsite.dk
X-Trace: 1702605607 news.sunsite.dk 719 116e32s@gmail.com/49.192.213.112:58000
X-Complaints-To: staff@sunsite.dk
 by: Robinn - Fri, 15 Dec 2023 01:59 UTC

I got some old neural network code (written about 30 years ago).
It has several activation functions, which only change 2 lines, like so:

if (activation(1:2).eq.'SI' .or. activation(1:2).eq.'LO') then
output(i,j) = 1.0/(1.0+EXP(-output(i,j))) ! sigmoid
slope(i,j) = output(i,j) * (1.0 - output(i,j)) ! sigmoid
elseif (activation(1:2).eq.'TA') then
output(i,j) = TANH(output(i,j)) ! TANH
slope(i,j) = 1.0 - output(i,j)*output(i,j) ! TANH
elseif (activation(1:2).eq.'AR') then
y = output(i,j)
output(i,j) = ATAN(y) ! arctan
slope(i,j) = 1.0/(1.0 +y*y) ! arctan
elseif (activation(1:5).eq.'SOFTP') then
y = EXP(output(i,j))
output(i,j) = LOG(1.0+y) ! softplus
slope(i,j) = 1.0/(1.0+1.0/y) ! softplus
elseif (activation(1:5).eq.'SOFTS') then
y = output(i,j)
output(i,j) = y/(ABS(y)+1.0) ! softsign
slope(i,j) = 1.0/(1.0+ABS(y))**2 ! softsign

Now when running it, the tanh option is slowest, as expected.
But the sigmoid (using exp) is faster than softsign, which only needs
abs and simple arithmetic. How can this be? Even if exp is using a
table lookup and spline interpolation, I would think that is slower.
Softsign would have an extra divide, but I can't see that tipping the
scales.

Re: CPU time for transcendental functions

<ulgk9l$1p5jm$1@dont-email.me>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=3783&group=comp.lang.fortran#3783

  copy link   Newsgroups: comp.lang.fortran
Path: i2pn2.org!i2pn.org!eternal-september.org!feeder3.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: sgk@REMOVEtroutmask.apl.washington.edu (Steven G. Kargl)
Newsgroups: comp.lang.fortran
Subject: Re: CPU time for transcendental functions
Date: Fri, 15 Dec 2023 04:22:13 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 37
Message-ID: <ulgk9l$1p5jm$1@dont-email.me>
References: <657bb327$0$719$14726298@news.sunsite.dk>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Injection-Date: Fri, 15 Dec 2023 04:22:13 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="47322a114c6a3dca9541b6169b0c40c5";
logging-data="1873526"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1//cojKWybccNVhVnbIXQxB"
User-Agent: Pan/0.145 (Duplicitous mercenary valetism; d7e168a
git.gnome.org/pan2)
Cancel-Lock: sha1:4/ySgZdHi9UWF+aXRpKs0/pVLCs=
 by: Steven G. Kargl - Fri, 15 Dec 2023 04:22 UTC

On Fri, 15 Dec 2023 09:59:56 +0800, Robinn wrote:

> I got some old neural network code (written about 30 years ago).
> It has several activation functions, which only change 2 lines, like so:
>
> if (activation(1:2).eq.'SI' .or. activation(1:2).eq.'LO') then
> output(i,j) = 1.0/(1.0+EXP(-output(i,j))) ! sigmoid
> slope(i,j) = output(i,j) * (1.0 - output(i,j)) ! sigmoid
> elseif (activation(1:2).eq.'TA') then
> output(i,j) = TANH(output(i,j)) ! TANH
> slope(i,j) = 1.0 - output(i,j)*output(i,j) ! TANH
> elseif (activation(1:2).eq.'AR') then
> y = output(i,j)
> output(i,j) = ATAN(y) ! arctan
> slope(i,j) = 1.0/(1.0 +y*y) ! arctan
> elseif (activation(1:5).eq.'SOFTP') then
> y = EXP(output(i,j))
> output(i,j) = LOG(1.0+y) ! softplus
> slope(i,j) = 1.0/(1.0+1.0/y) ! softplus
> elseif (activation(1:5).eq.'SOFTS') then
> y = output(i,j)
> output(i,j) = y/(ABS(y)+1.0) ! softsign
> slope(i,j) = 1.0/(1.0+ABS(y))**2 ! softsign
>
> Now when running it, the tanh option is slowest, as expected.
> But the sigmoid (using exp) is faster than softsign, which only needs
> abs and simple arithmetic. How can this be? Even if exp is using a
> table lookup and spline interpolation, I would think that is slower.
> Softsign would have an extra divide, but I can't see that tipping the
> scales.

There is insufficient information to provide much help. First, what
compiler and operating system? Second, how did you do the timing?
Third, is there a minimum working example that others can profile?

--
steve

Re: CPU time for transcendental functions

<kulla0Fu4ebU2@mid.individual.net>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=3788&group=comp.lang.fortran#3788

  copy link   Newsgroups: comp.lang.fortran
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail
From: pastgio@units.it (Giorgio Pastore)
Newsgroups: comp.lang.fortran
Subject: Re: CPU time for transcendental functions
Date: Fri, 22 Dec 2023 15:37:52 +0100
Lines: 40
Message-ID: <kulla0Fu4ebU2@mid.individual.net>
References: <657bb327$0$719$14726298@news.sunsite.dk>
<ulgk9l$1p5jm$1@dont-email.me>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Trace: individual.net DUiYZ7ytSwykK/wWvPBMIgCx79GK4pNiKQhsOFT5muBaOOmLBO
Cancel-Lock: sha1:Vnmih2ehuTq1aP2pvVJDFaW7RQs= sha256:qgHfv69FdfT4G1E2F9XlcZToEg8bAz4mOj2Mm3Ums28=
User-Agent: Mozilla Thunderbird
Content-Language: it
In-Reply-To: <ulgk9l$1p5jm$1@dont-email.me>
 by: Giorgio Pastore - Fri, 22 Dec 2023 14:37 UTC

Il 15/12/23 05:22, Steven G. Kargl ha scritto:
> On Fri, 15 Dec 2023 09:59:56 +0800, Robinn wrote:
>
>> I got some old neural network code (written about 30 years ago).
>> It has several activation functions, which only change 2 lines, like so:
>>
>> if (activation(1:2).eq.'SI' .or. activation(1:2).eq.'LO') then
>> output(i,j) = 1.0/(1.0+EXP(-output(i,j))) ! sigmoid
>> slope(i,j) = output(i,j) * (1.0 - output(i,j)) ! sigmoid
>> elseif (activation(1:2).eq.'TA') then
>> output(i,j) = TANH(output(i,j)) ! TANH
>> slope(i,j) = 1.0 - output(i,j)*output(i,j) ! TANH
>> elseif (activation(1:2).eq.'AR') then
>> y = output(i,j)
>> output(i,j) = ATAN(y) ! arctan
>> slope(i,j) = 1.0/(1.0 +y*y) ! arctan
>> elseif (activation(1:5).eq.'SOFTP') then
>> y = EXP(output(i,j))
>> output(i,j) = LOG(1.0+y) ! softplus
>> slope(i,j) = 1.0/(1.0+1.0/y) ! softplus
>> elseif (activation(1:5).eq.'SOFTS') then
>> y = output(i,j)
>> output(i,j) = y/(ABS(y)+1.0) ! softsign
>> slope(i,j) = 1.0/(1.0+ABS(y))**2 ! softsign
>>
>> Now when running it, the tanh option is slowest, as expected.
>> But the sigmoid (using exp) is faster than softsign, which only needs
>> abs and simple arithmetic. How can this be? Even if exp is using a
>> table lookup and spline interpolation, I would think that is slower.
>> Softsign would have an extra divide, but I can't see that tipping the
>> scales.
>
> There is insufficient information to provide much help. First, what
> compiler and operating system? Second, how did you do the timing?
> Third, is there a minimum working example that others can profile?
>

Fourth, what were the numbers of timing.

Giorgio

Re: CPU time for transcendental functions

<upacll$tq89$1@dont-email.me>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=3847&group=comp.lang.fortran#3847

  copy link   Newsgroups: comp.lang.fortran
Path: i2pn2.org!i2pn.org!eternal-september.org!feeder3.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: jahns@idontlikespam.dkrz.de (Thomas Jahns)
Newsgroups: comp.lang.fortran
Subject: Re: CPU time for transcendental functions
Date: Tue, 30 Jan 2024 09:40:22 +0100
Organization: A noiseless patient Spider
Lines: 35
Message-ID: <upacll$tq89$1@dont-email.me>
References: <657bb327$0$719$14726298@news.sunsite.dk>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Tue, 30 Jan 2024 08:40:21 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="a6fba6b5028031c5be38736e738d6857";
logging-data="977161"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX185GZs9gvDmQqriSWzKkAQd"
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
Thunderbird/68.10.0
Cancel-Lock: sha1:ENoLhZXoPItH5M5W7c7U5mCo2/g=
Content-Language: en-US
In-Reply-To: <657bb327$0$719$14726298@news.sunsite.dk>
 by: Thomas Jahns - Tue, 30 Jan 2024 08:40 UTC

On 2023-12-15 02:59, Robinn wrote:
> I got some old neural network code (written about 30 years ago).
> It has several activation functions, which only change 2 lines, like so:
>
>       if (activation(1:2).eq.'SI' .or. activation(1:2).eq.'LO') then
>          output(i,j) = 1.0/(1.0+EXP(-output(i,j)))       ! sigmoid
>          slope(i,j) = output(i,j) * (1.0 - output(i,j)) ! sigmoid
>       elseif (activation(1:2).eq.'TA') then
>          output(i,j) = TANH(output(i,j))                 ! TANH
>          slope(i,j) = 1.0 - output(i,j)*output(i,j)     ! TANH
>       elseif (activation(1:2).eq.'AR') then
>          y = output(i,j)
>          output(i,j) = ATAN(y)                           ! arctan
>          slope(i,j) = 1.0/(1.0 +y*y)                  ! arctan
>       elseif (activation(1:5).eq.'SOFTP') then
>          y = EXP(output(i,j))
>          output(i,j) = LOG(1.0+y)                        ! softplus
>          slope(i,j) = 1.0/(1.0+1.0/y)               ! softplus
>       elseif (activation(1:5).eq.'SOFTS') then
>          y = output(i,j)
>          output(i,j) = y/(ABS(y)+1.0)                    ! softsign
>          slope(i,j) = 1.0/(1.0+ABS(y))**2             ! softsign
>
> Now when running it, the tanh option is slowest, as expected.
> But the sigmoid (using exp) is faster than softsign, which only needs
> abs  and simple arithmetic. How can this be? Even if exp is using a table lookup
> and spline interpolation, I would think that is slower.
> Softsign would have an extra divide, but I can't see that tipping the scales.

You perhaps are not aware that the string comparisons (for which most compilers
call the strncmp function) you have in your conditionals are quite expensive on
todays CPUs. I would recommend to use an INTEGER constant to make the switch.

Thomas

1
server_pubkey.txt

rocksolid light 0.9.81
clearnet tor