Hi!

Inserts through the Go wrapper are about 10 times slower comparing with M.
By inserting 1M of values.

time ./speedYotta

real 0m13,728s
user 0m18,737s
sys 0m0,914s

time yottadb -run ^speedYotta

real 0m1,617s
user 0m1,326s
sys 0m0,278s

time ./sY1

real 0m13,075s
user 0m18,786s
sys 0m0,751s

Source codes:
=== speedYotta.go ===
-------------------
package main

import (
"lang.yottadb.com/go/yottadb"
"strconv"
)

func main() {

defer yottadb.Exit()

s := ""
for i := 1; i < 1000000; i++ {
s = strconv.Itoa(i)
yottadb.SetValE(yottadb.NOTTP, nil, s, "^hello", []string{s})
}
} -------------------------

=== speedYotta.m ===
---------------------------
speedYotta

for i=0:1:1000000 do
. set ^a(i)=i
---------------------------

=== sY1.go ===
---------------------------
package main

import (
"lang.yottadb.com/go/yottadb"
"strconv"
)

func main() {
defer yottadb.Exit()

s := strconv.Itoa(5)
for i := 1; i < 1000000; i++ {
yottadb.SetValE(yottadb.NOTTP, nil, s, "^hello", []string{s})
}
} ---------------------------

On Monday, October 25, 2021 at 1:47:58 PM UTC-4, suka...@gmail.com wrote:
> Hi!
>
> Inserts through the Go wrapper are about 10 times slower comparing with M..
> By inserting 1M of values.
>
> time ./speedYotta
>
> real 0m13,728s
> user 0m18,737s
> sys 0m0,914s
>
> time yottadb -run ^speedYotta
>
> real 0m1,617s
> user 0m1,326s
> sys 0m0,278s
>
> time ./sY1
>
> real 0m13,075s
> user 0m18,786s
> sys 0m0,751s
>
> Source codes:
> === speedYotta.go ===
> -------------------
> package main
>
> import (
> "lang.yottadb.com/go/yottadb"
> "strconv"
> )
>
> func main() {
>
> defer yottadb.Exit()
>
> s := ""
> for i := 1; i < 1000000; i++ {
> s = strconv.Itoa(i)
> yottadb.SetValE(yottadb.NOTTP, nil, s, "^hello", []string{s})
> }
> }
> -------------------------
>
> === speedYotta.m ===
> ---------------------------
> speedYotta
>
> for i=0:1:1000000 do
> . set ^a(i)=i
> ---------------------------
>
> === sY1.go ===
> ---------------------------
> package main
>
> import (
> "lang.yottadb.com/go/yottadb"
> "strconv"
> )
>
> func main() {
> defer yottadb.Exit()
>
> s := strconv.Itoa(5)
> for i := 1; i < 1000000; i++ {
> yottadb.SetValE(yottadb.NOTTP, nil, s, "^hello", []string{s})
> }
> }
> ---------------------------

There are two reasons. The biggest one is that CGO is heavyweight. A much smaller one is the way Go does garbage collection emphasizes predictable response time, the way that YottaDB does garbage collection is to minimize CPU usage. You are basically doing what I'd call a “point” benchmark. In the context of a complete application, where an application executes business logic interspersed with database accesses, you are likely to find different performance characteristics. For example, Go's applications using Goroutines may execute business logic faster than M's single flow-of-control code execution.

Regards
– Bhaskar

Is it potentially possible to increase the speed of the CGO wrapper?

Sergey.

> > Hi!
> >
> > Inserts through the Go wrapper are about 10 times slower comparing with M.
> > By inserting 1M of values.
> >
> > time ./speedYotta
> >
> > real 0m13,728s
> > user 0m18,737s
> > sys 0m0,914s
> >
> > time yottadb -run ^speedYotta
> >
> > real 0m1,617s
> > user 0m1,326s
> > sys 0m0,278s
> >
> > time ./sY1
> >
> > real 0m13,075s
> > user 0m18,786s
> > sys 0m0,751s
> >
> > Source codes:
> > === speedYotta.go ===
> > -------------------
> > package main
> >
> > import (
> > "lang.yottadb.com/go/yottadb"
> > "strconv"
> > )
> >
> > func main() {
> >
> > defer yottadb.Exit()
> >
> > s := ""
> > for i := 1; i < 1000000; i++ {
> > s = strconv.Itoa(i)
> > yottadb.SetValE(yottadb.NOTTP, nil, s, "^hello", []string{s})
> > }
> > }
> > -------------------------
> >
> > === speedYotta.m ===
> > ---------------------------
> > speedYotta
> >
> > for i=0:1:1000000 do
> > . set ^a(i)=i
> > ---------------------------
> >
> > === sY1.go ===
> > ---------------------------
> > package main
> >
> > import (
> > "lang.yottadb.com/go/yottadb"
> > "strconv"
> > )
> >
> > func main() {
> > defer yottadb.Exit()
> >
> > s := strconv.Itoa(5)
> > for i := 1; i < 1000000; i++ {
> > yottadb.SetValE(yottadb.NOTTP, nil, s, "^hello", []string{s})
> > }
> > }
> > ---------------------------
> There are two reasons. The biggest one is that CGO is heavyweight. A much smaller one is the way Go does garbage collection emphasizes predictable response time, the way that YottaDB does garbage collection is to minimize CPU usage. You are basically doing what I'd call a “point” benchmark. In the context of a complete application, where an application executes business logic interspersed with database accesses, you are likely to find different performance characteristics. For example, Go's applications using Goroutines may execute business logic faster than M's single flow-of-control code execution.
>
> Regards
> – Bhaskar

On Monday, October 25, 2021 at 3:00:19 PM UTC-4, suka...@gmail.com wrote:
> Is it potentially possible to increase the speed of the CGO wrapper?
>
> Sergey.
> > > Hi!
> > >
> > > Inserts through the Go wrapper are about 10 times slower comparing with M.
> > > By inserting 1M of values.
> > >
> > > time ./speedYotta
> > >
> > > real 0m13,728s
> > > user 0m18,737s
> > > sys 0m0,914s
> > >
> > > time yottadb -run ^speedYotta
> > >
> > > real 0m1,617s
> > > user 0m1,326s
> > > sys 0m0,278s
> > >
> > > time ./sY1
> > >
> > > real 0m13,075s
> > > user 0m18,786s
> > > sys 0m0,751s
> > >
> > > Source codes:
> > > === speedYotta.go ===
> > > -------------------
> > > package main
> > >
> > > import (
> > > "lang.yottadb.com/go/yottadb"
> > > "strconv"
> > > )
> > >
> > > func main() {
> > >
> > > defer yottadb.Exit()
> > >
> > > s := ""
> > > for i := 1; i < 1000000; i++ {
> > > s = strconv.Itoa(i)
> > > yottadb.SetValE(yottadb.NOTTP, nil, s, "^hello", []string{s})
> > > }
> > > }
> > > -------------------------
> > >
> > > === speedYotta.m ===
> > > ---------------------------
> > > speedYotta
> > >
> > > for i=0:1:1000000 do
> > > . set ^a(i)=i
> > > ---------------------------
> > >
> > > === sY1.go ===
> > > ---------------------------
> > > package main
> > >
> > > import (
> > > "lang.yottadb.com/go/yottadb"
> > > "strconv"
> > > )
> > >
> > > func main() {
> > > defer yottadb.Exit()
> > >
> > > s := strconv.Itoa(5)
> > > for i := 1; i < 1000000; i++ {
> > > yottadb.SetValE(yottadb.NOTTP, nil, s, "^hello", []string{s})
> > > }
> > > }
> > > ---------------------------
> > There are two reasons. The biggest one is that CGO is heavyweight. A much smaller one is the way Go does garbage collection emphasizes predictable response time, the way that YottaDB does garbage collection is to minimize CPU usage. You are basically doing what I'd call a “point” benchmark. In the context of a complete application, where an application executes business logic interspersed with database accesses, you are likely to find different performance characteristics. For example, Go's applications using Goroutines may execute business logic faster than M's single flow-of-control code execution.
> >
> > Regards
> > – Bhaskar

There has been much discussion of CGO overhead on the golang-nuts discussion group.

Incidentally, you are using the YottaDB Go Easy API. You may find the Go Simple API to be a bit faster, although I don't know how much faster it will be for you.

Regards
– Bhaskar

> > Is it potentially possible to increase the speed of the CGO wrapper?
> >
> > Sergey.
> > > > Hi!
> > > >
> > > > Inserts through the Go wrapper are about 10 times slower comparing with M.
> > > > By inserting 1M of values.
> > > >
> > > > time ./speedYotta
> > > >
> > > > real 0m13,728s
> > > > user 0m18,737s
> > > > sys 0m0,914s
> > > >
> > > > time yottadb -run ^speedYotta
> > > >
> > > > real 0m1,617s
> > > > user 0m1,326s
> > > > sys 0m0,278s
> > > >
> > > > time ./sY1
> > > >
> > > > real 0m13,075s
> > > > user 0m18,786s
> > > > sys 0m0,751s
> > > >
> > > > Source codes:
> > > > === speedYotta.go ===
> > > > -------------------
> > > > package main
> > > >
> > > > import (
> > > > "lang.yottadb.com/go/yottadb"
> > > > "strconv"
> > > > )
> > > >
> > > > func main() {
> > > >
> > > > defer yottadb.Exit()
> > > >
> > > > s := ""
> > > > for i := 1; i < 1000000; i++ {
> > > > s = strconv.Itoa(i)
> > > > yottadb.SetValE(yottadb.NOTTP, nil, s, "^hello", []string{s})
> > > > }
> > > > }
> > > > -------------------------
> > > >
> > > > === speedYotta.m ===
> > > > ---------------------------
> > > > speedYotta
> > > >
> > > > for i=0:1:1000000 do
> > > > . set ^a(i)=i
> > > > ---------------------------
> > > >
> > > > === sY1.go ===
> > > > ---------------------------
> > > > package main
> > > >
> > > > import (
> > > > "lang.yottadb.com/go/yottadb"
> > > > "strconv"
> > > > )
> > > >
> > > > func main() {
> > > > defer yottadb.Exit()
> > > >
> > > > s := strconv.Itoa(5)
> > > > for i := 1; i < 1000000; i++ {
> > > > yottadb.SetValE(yottadb.NOTTP, nil, s, "^hello", []string{s})
> > > > }
> > > > }
> > > > ---------------------------
> > > There are two reasons. The biggest one is that CGO is heavyweight. A much smaller one is the way Go does garbage collection emphasizes predictable response time, the way that YottaDB does garbage collection is to minimize CPU usage. You are basically doing what I'd call a “point” benchmark. In the context of a complete application, where an application executes business logic interspersed with database accesses, you are likely to find different performance characteristics. For example, Go's applications using Goroutines may execute business logic faster than M's single flow-of-control code execution.
> > >
> > > Regards
> > > – Bhaskar
> There has been much discussion of CGO overhead on the golang-nuts discussion group.
>
> Incidentally, you are using the YottaDB Go Easy API. You may find the Go Simple API to be a bit faster, although I don't know how much faster it will be for you.
>
> Regards
> – Bhaskar

Thank you, Bhaskar!

Yes, Go Simple API 3 times faster comparing with Go Easy API.
And Go Simple API 3 times slower comparing with native M code for inserting..

time ./speedYSimple

real 0m4,391s
user 0m4,250s
sys 0m0,570s

If we move some operations per cycle, then

time ./speedYSimple1

real 0m3,425s
user 0m3,296s
sys 0m0,458s

=== Code of speedYSimple.go: ====
package main

import (
"lang.yottadb.com/go/yottadb"
"strconv"
)

const maxglnamelen uint32 = 8
const maxglsubscount uint32 = 1
const maxglsubslen uint32 = 10 //128
const tptoken uint64 = yottadb.NOTTP

func main() {
var glname yottadb.KeyT
var errstr, value yottadb.BufferT

value.Alloc(maxglsubslen)
errstr.Alloc(yottadb.YDB_MAX_ERRORMSG)
glname.Alloc(maxglnamelen, maxglsubscount, maxglsubslen)

defer yottadb.Exit()

defer glname.Free()
defer errstr.Free()
defer value.Free()

s := ""
for i := 1; i < 1000000; i++ {
s = strconv.Itoa(i)

glname.Varnm.SetValStr(tptoken, &errstr, "^hello")
glname.Subary.SetElemUsed(tptoken, &errstr, maxglsubscount)
glname.Subary.SetValStr(tptoken, &errstr, 0, s)

value.SetValStr(tptoken, &errstr, s)

glname.SetValST(tptoken, &errstr, &value)
}
}

=== Code of speedYSimple1.go: ====
package main

import (
"lang.yottadb.com/go/yottadb"
"strconv"
)

const maxglnamelen uint32 = 8
const maxglsubscount uint32 = 1
const maxglsubslen uint32 = 10 //128
const tptoken uint64 = yottadb.NOTTP

func main() {
var glname yottadb.KeyT
var errstr, value yottadb.BufferT

value.Alloc(maxglsubslen)
errstr.Alloc(yottadb.YDB_MAX_ERRORMSG)
glname.Alloc(maxglnamelen, maxglsubscount, maxglsubslen)

defer yottadb.Exit()

defer glname.Free()
defer errstr.Free()
defer value.Free()

glname.Varnm.SetValStr(tptoken, &errstr, "^hello")
glname.Subary.SetElemUsed(tptoken, &errstr, maxglsubscount)

s := ""
for i := 1; i < 1000000; i++ {
s = strconv.Itoa(i)

glname.Subary.SetValStr(tptoken, &errstr, 0, s)

value.SetValStr(tptoken, &errstr, s)

glname.SetValST(tptoken, &errstr, &value)
}
}

On Tuesday, October 26, 2021 at 10:04:30 AM UTC-4, suka...@gmail.com wrote:
> > > Is it potentially possible to increase the speed of the CGO wrapper?
> > >
> > > Sergey.
> > > > > Hi!
> > > > >
> > > > > Inserts through the Go wrapper are about 10 times slower comparing with M.
> > > > > By inserting 1M of values.
> > > > >
> > > > > time ./speedYotta
> > > > >
> > > > > real 0m13,728s
> > > > > user 0m18,737s
> > > > > sys 0m0,914s
> > > > >
> > > > > time yottadb -run ^speedYotta
> > > > >
> > > > > real 0m1,617s
> > > > > user 0m1,326s
> > > > > sys 0m0,278s
> > > > >
> > > > > time ./sY1
> > > > >
> > > > > real 0m13,075s
> > > > > user 0m18,786s
> > > > > sys 0m0,751s
> > > > >
> > > > > Source codes:
> > > > > === speedYotta.go ===
> > > > > -------------------
> > > > > package main
> > > > >
> > > > > import (
> > > > > "lang.yottadb.com/go/yottadb"
> > > > > "strconv"
> > > > > )
> > > > >
> > > > > func main() {
> > > > >
> > > > > defer yottadb.Exit()
> > > > >
> > > > > s := ""
> > > > > for i := 1; i < 1000000; i++ {
> > > > > s = strconv.Itoa(i)
> > > > > yottadb.SetValE(yottadb.NOTTP, nil, s, "^hello", []string{s})
> > > > > }
> > > > > }
> > > > > -------------------------
> > > > >
> > > > > === speedYotta.m ===
> > > > > ---------------------------
> > > > > speedYotta
> > > > >
> > > > > for i=0:1:1000000 do
> > > > > . set ^a(i)=i
> > > > > ---------------------------
> > > > >
> > > > > === sY1.go ===
> > > > > ---------------------------
> > > > > package main
> > > > >
> > > > > import (
> > > > > "lang.yottadb.com/go/yottadb"
> > > > > "strconv"
> > > > > )
> > > > >
> > > > > func main() {
> > > > > defer yottadb.Exit()
> > > > >
> > > > > s := strconv.Itoa(5)
> > > > > for i := 1; i < 1000000; i++ {
> > > > > yottadb.SetValE(yottadb.NOTTP, nil, s, "^hello", []string{s})
> > > > > }
> > > > > }
> > > > > ---------------------------
> > > > There are two reasons. The biggest one is that CGO is heavyweight. A much smaller one is the way Go does garbage collection emphasizes predictable response time, the way that YottaDB does garbage collection is to minimize CPU usage. You are basically doing what I'd call a “point” benchmark. In the context of a complete application, where an application executes business logic interspersed with database accesses, you are likely to find different performance characteristics. For example, Go's applications using Goroutines may execute business logic faster than M's single flow-of-control code execution.
> > > >
> > > > Regards
> > > > – Bhaskar
> > There has been much discussion of CGO overhead on the golang-nuts discussion group.
> >
> > Incidentally, you are using the YottaDB Go Easy API. You may find the Go Simple API to be a bit faster, although I don't know how much faster it will be for you.
> >
> > Regards
> > – Bhaskar
> Thank you, Bhaskar!
>
> Yes, Go Simple API 3 times faster comparing with Go Easy API.
> And Go Simple API 3 times slower comparing with native M code for inserting.
>
> time ./speedYSimple
>
> real 0m4,391s
> user 0m4,250s
> sys 0m0,570s
>
> If we move some operations per cycle, then
>
> time ./speedYSimple1
>
> real 0m3,425s
> user 0m3,296s
> sys 0m0,458s
>
> === Code of speedYSimple.go: ====> package main
>
> import (
> "lang.yottadb.com/go/yottadb"
> "strconv"
> )
> const maxglnamelen uint32 = 8
> const maxglsubscount uint32 = 1
> const maxglsubslen uint32 = 10 //128
> const tptoken uint64 = yottadb.NOTTP
>
> func main() {
> var glname yottadb.KeyT
> var errstr, value yottadb.BufferT
>
> value.Alloc(maxglsubslen)
> errstr.Alloc(yottadb.YDB_MAX_ERRORMSG)
> glname.Alloc(maxglnamelen, maxglsubscount, maxglsubslen)
>
> defer yottadb.Exit()
>
> defer glname.Free()
> defer errstr.Free()
> defer value.Free()
> s := ""
> for i := 1; i < 1000000; i++ {
> s = strconv.Itoa(i)
> glname.Varnm.SetValStr(tptoken, &errstr, "^hello")
> glname.Subary.SetElemUsed(tptoken, &errstr, maxglsubscount)
> glname.Subary.SetValStr(tptoken, &errstr, 0, s)
>
> value.SetValStr(tptoken, &errstr, s)
>
> glname.SetValST(tptoken, &errstr, &value)
> }
> }
>
>
> === Code of speedYSimple1.go: ====> package main
>
> import (
> "lang.yottadb.com/go/yottadb"
> "strconv"
> )
> const maxglnamelen uint32 = 8
> const maxglsubscount uint32 = 1
> const maxglsubslen uint32 = 10 //128
> const tptoken uint64 = yottadb.NOTTP
>
> func main() {
> var glname yottadb.KeyT
> var errstr, value yottadb.BufferT
>
> value.Alloc(maxglsubslen)
> errstr.Alloc(yottadb.YDB_MAX_ERRORMSG)
> glname.Alloc(maxglnamelen, maxglsubscount, maxglsubslen)
>
> defer yottadb.Exit()
>
> defer glname.Free()
> defer errstr.Free()
> defer value.Free()
>
> glname.Varnm.SetValStr(tptoken, &errstr, "^hello")
> glname.Subary.SetElemUsed(tptoken, &errstr, maxglsubscount)
> s := ""
> for i := 1; i < 1000000; i++ {
> s = strconv.Itoa(i)
> glname.Subary.SetValStr(tptoken, &errstr, 0, s)
>
> value.SetValStr(tptoken, &errstr, s)
>
> glname.SetValST(tptoken, &errstr, &value)
> }
> }

Thanks for the update. I'm glad you found the simple API to be faster.

Regards
– Bhaskar

Very good speed and very simple interface have "mg_go" extension.
https://github.com/chrisemunt/mg_go

I'm not sure if he knows how to handle Go-multithreading and parallel transactions correctly. But using it is very simple and pleasant.

time ./speedYMG

real 0m3,196s
user 0m3,069s
sys 0m0,326s

=== Code speedYMG.go ===
package main

import (
"mg_go"
)

func main() {
db := mg_go.New("YottaDB")
db.APImodule = "/usr/local/lib64/mg_dba.so"
db.Path = "/opt/yottadb/r1.32"
db.EnvVars = db.EnvVars + "ydb_dir=/home/inetstar/.yottadb\n"
db.EnvVars = db.EnvVars + "ydb_rel=r1.32_x86_64\n"
db.EnvVars = db.EnvVars + "ydb_gbldir=/home/inetstar/.yottadb/r1.32_x86_64/g/yottadb.gld\n"
db.EnvVars = db.EnvVars + "ydb_routines=/home/inetstar/.yottadb/r1.32_x86_64/o*(/root/.yottadb/r1.32_x86_64/r /home/inetstar/.yottadb/r) /opt/yottadb/r1.32/libyottadbutil.so\n"
db.EnvVars = db.EnvVars + "ydb_ci=/opt/yottadb/r1.32/zmgci.ci\n"
db.EnvVars = db.EnvVars + "\n"

db.Open()

defer db.Close()

gHello := db.Global("^hello")

for i := 1; i < 1000000; i++ {
gHello.Set(i, i)
}
}

On Wednesday, October 27, 2021 at 9:45:56 AM UTC-4, suka...@gmail.com wrote:
> Very good speed and very simple interface have "mg_go" extension.
> https://github.com/chrisemunt/mg_go
>
> I'm not sure if he knows how to handle Go-multithreading and parallel transactions correctly. But using it is very simple and pleasant.
>
> time ./speedYMG
>
> real 0m3,196s
> user 0m3,069s
> sys 0m0,326s
>
> === Code speedYMG.go ===
> package main
>
> import (
> "mg_go"
> )
>
> func main() {
> db := mg_go.New("YottaDB")
> db.APImodule = "/usr/local/lib64/mg_dba.so"
> db.Path = "/opt/yottadb/r1.32"
> db.EnvVars = db.EnvVars + "ydb_dir=/home/inetstar/.yottadb\n"
> db.EnvVars = db.EnvVars + "ydb_rel=r1.32_x86_64\n"
> db.EnvVars = db.EnvVars + "ydb_gbldir=/home/inetstar/.yottadb/r1.32_x86_64/g/yottadb.gld\n"
> db.EnvVars = db.EnvVars + "ydb_routines=/home/inetstar/.yottadb/r1.32_x86_64/o*(/root/.yottadb/r1.32_x86_64/r /home/inetstar/.yottadb/r) /opt/yottadb/r1.32/libyottadbutil.so\n"
> db.EnvVars = db.EnvVars + "ydb_ci=/opt/yottadb/r1.32/zmgci.ci\n"
> db.EnvVars = db.EnvVars + "\n"
>
> db.Open()
>
> defer db.Close()
>
> gHello := db.Global("^hello")
> for i := 1; i < 1000000; i++ {
> gHello.Set(i, i)
> }
> }

That's good news, and if that works for you, please use it. Chris Munt does good work. The only thing to check is to make sure that it works well with Goroutines if your application uses them, as most applications do.

Regards
– Bhaskar