When some/all com/exe receive control from MSDOS,
all available memory has been allocated and the onus
is on the executable to free it if it wishes to be able to
do some mallocs in the future.

What is the rationale for this, and what is the situation?

I have startup code below that resizes that memory, as
per rules I no longer remember. I'd like to remove this
code if possible because it hardcodes the segment
shift (4), and it prevents the code/data/stack being
placed in non-contiguous areas of memory.

Where do I stand?

Thanks. Paul.

; determine how much memory is needed. The stack pointer points
; to the top. Work out what segment that is, then subtract the
; starting segment (the PSP), and you have your answer.

mov ax, sp
mov cl, 4
shr ax, cl ; get sp into pages
mov bx, ss
add ax, bx
add ax, 2 ; safety margin because we've done some pushes etc
mov bx, es
sub ax, bx ; subtract the psp segment

; free initially allocated memory

mov bx, ax
mov ah, 4ah
int 21h

On Thu, 2 Dec 2021 14:07:04 -0800 (PST), muta...@gmail.com wrote:
> When some/all com/exe receive control from MSDOS,
> all available memory has been allocated and the onus
> is on the executable to free it if it wishes to be able to
> do some mallocs in the future.
>
> What is the rationale for this, and what is the situation?

It's for backward compatibility reason. Keep in mind that, MCB is added only
at later DOS version.

> I have startup code below that resizes that memory, as
> per rules I no longer remember. I'd like to remove this
> code if possible because it hardcodes the segment
> shift (4), and it prevents the code/data/stack being
> placed in non-contiguous areas of memory.
>
> Where do I stand?
>
> Thanks. Paul.

That code shouldn't removed for TSRs and programs which need to spawn other
program. If a program needs continuous memory area, it needs to do so
manually.

MCB is not meant to separate programs' parts such as initial data, stacks,
heaps, buffers, etc. In DOS, those are application managed. As for stack,
DOS simply prepares the memory area for it. It's up to the application what
to do with it, once DOS gives control to the application.

On Friday, December 3, 2021 at 7:06:01 PM UTC+11, JJ wrote:

> That code shouldn't removed for TSRs and programs which need to spawn other
> program.

Ok, this is generic startup code for C programs, which may
do a system().

So for all executables, all of memory is allocated until
released?

Do you think that ideally MSDOS 2.0 should have included
a call to auto-release extra memory?

PDOS/86 has the ability to add extra calls like that. But I
need some way of testing whether I am running under a
PDOS-compliant (or some other existing standard). I was
thinking of getting the MSDOS version number and if it
was 0.1 or something like that, then it meant additional
calls were available to find more things like manufacturer
and capability (extra INT 21H calls).

BFN. Paul.

2021-12-03 at 00:38 -0800, muta...@gmail.com wrote:
> So for all executables, all of memory is allocated until
> released?

AFAIK this is true only for COM files, where CS=DS=ES=SS.
This mechanism allows for a very easy way DOS can load these programs.
Just allocate a single 64K block, put the COM image there, prefixed
with a PSP, preset CS=DS=ES=SS and point SP to the end of block.
EXE-style loading, in contrast, is *way* more complex.

> Do you think that ideally MSDOS 2.0 should have included
> a call to auto-release extra memory?

There is no reliable way to know how much memory your program needs.
Only you know that.

Mateusz

On Friday, December 3, 2021 at 7:45:01 PM UTC+11, Mateusz Viste wrote:

> > Do you think that ideally MSDOS 2.0 should have included
> > a call to auto-release extra memory?

> There is no reliable way to know how much memory your program needs.
> Only you know that.

It looks like this is the problem:

https://stackoverflow.com/questions/43214906/asm-exe-program-16bit-error-changing-size-of-memory

tlink (which is what I originally used) puts x'ffff' as the number
of paragraphs to allocate, meaning allocate all memory.

So this problem needs to be solved by me using a sensible
toolchain.

I still need to know whether PDOS extensions are available
though, because I want to be able to allocate memory by a
32-bit number of bytes rather than paragraphs (and returns
a far pointer), so need another API call. Because I want to
produce an executable that works on both RM16 and PM16
with linear GDT+LDT.

BFN. Paul.

Muta,

> I have startup code below that resizes that memory, as
> per rules I no longer remember.
....
> mov ax, sp
> mov cl, 4
> shr ax, cl ; get sp into pages

When a COM runs SP is set to the end of the SS segment. As a COM has all
the segments overlapping its upto how big your program is how much stack you
are actually reserving. But looking at my own programs I don't think that
reserving 60KB stack for a 4KB program is normally called for.

Hence I set SP myself (sometimes even to the programs top at 0100h) and
instead use a LASTBYTE label to figure out how much room I actually need
calculated the # of pages from it :

- - - - - - - - - - - - - - - - - - - -
lea sp,[StackTop] ;Set new Stack-top

mov bx,(offset LASTBYTE)+000Fh ;number of bytes in program + round-up
mov cl,04h ;convert to pages
shr bx,cl ;/
mov ah,4ah ;resize program space
int 21h ;/
jc @@Main_Error ;ERROR : Out of Memory !
- - - - - - - - - - - - - - - - - - - -

As you might see, I also made sure I could actually allocate it, as its
possible a COM gets loaded in a memory fragment thats not actually large
enough to hold it and the "extra" memory it needs.

> So for all executables, all of memory is allocated until
> released?

Yes for a .COM style program, and "it depends" for a .EXE style one - as
that latter one has got a header wich specifies lots of stuff.

> tlink (which is what I originally used) puts x'ffff' as the number
> of paragraphs to allocate, meaning allocate all memory.

Indeed it does (using Borlands Tasm v4.1 and TLink v7.1). I've never been
able to find a "minimize footprint" switch or setting, which has irked me to
no end. You see, one of the fields in an .EXEs header is
"min_extra_paragraphs" , containing - you guessed it - exactly how much it
needs. As such I wrote a small program that copies that field into the
"max_extra_paragraphs" one (which is, in our case, the culprit), recalculate
the SS:SP value from it and put it into their respective fields.

From than on the OS does, for the .EXE, all the work for me, allowing me to
drop the "shrink my memory" code from it. :-)

Regards,
Rudy Wieser

On Friday, December 3, 2021 at 9:34:34 PM UTC+11, R.Wieser wrote:

> > tlink (which is what I originally used) puts x'ffff' as the number
> > of paragraphs to allocate, meaning allocate all memory.

> Indeed it does (using Borlands Tasm v4.1 and TLink v7.1). I've never been
> able to find a "minimize footprint" switch or setting, which has irked me to
> no end. You see, one of the fields in an .EXEs header is
> "min_extra_paragraphs" , containing - you guessed it - exactly how much it
> needs. As such I wrote a small program that copies that field into the
> "max_extra_paragraphs" one (which is, in our case, the culprit), recalculate
> the SS:SP value from it and put it into their respective fields.

Could you explain these min/max/ss:sp rules please?

I see from here:

http://blog.marcinchwedczuk.pl/a-closer-look-at-portable-executable-msdos-stub

e_cp: 0x0003 // Pages in file

e_minalloc: 0x0000 // Minimum extra paragraphs needed
e_maxalloc: 0xffff // Maximum extra paragraphs needed

e_ss: 0x0000 // Initial (relative) SS value
e_sp: 0x00b8 // Initial SP value

What I would guess is that e_minalloc is the size of the
bss and stack segment. e_cp is the size of the code and
data segments (not including bss).

And thus I don't know why you can't just copy the minalloc
value to the maxalloc.

Or maybe minalloc is ONLY the bss size, in which case the
maxalloc extends the BSS, and the stack is placed after that.
That would be surprising though, as it means when you
release the memory you create a hole. But maybe the
allocated memory doesn't include the stack in the first place.

Looking at the PDOS/86 code it looks like the stack size is
included, and you can't create a hole otherwise your stack
will be overwritten by a spawned program.

BFN. Paul.

Mta (paul),

> Could you explain these min/max/ss:sp rules please?

Perhaps. Its a while ago ...

> What I would guess is that e_minalloc is the size of the
> bss and stack segment.

Yup.

> And thus I don't know why you can't just copy the minalloc
> value to the maxalloc.

Thats exactly what I did.

But when you do that you also need to recalculate SS/SP, otherwise it most
likely will point far beyond your program. Which is not a good idea.

> Looking at the PDOS/86 code it looks like the stack size is
> included,

Yep. Its comprised of all data areas that have no ititial value.

> and you can't create a hole otherwise your stack
> will be overwritten by a spawned program.

I think we need to define what a "hole" is here. But yes, its never a good
idea use memory areas you do not own for either BSS or stack.

> Or maybe minalloc is ONLY the bss size

Definitily not.

But why don't you just try to make a program in which you enlarge the code,
BSS and stack areas (several combinations) and look at how the fields in the
EXE header change ? I found the EXE header info on the web and that is
what I did to verify if I understood it right.

Regards,
Rudy Wieser

On Saturday, December 4, 2021 at 12:20:05 AM UTC+11, R.Wieser wrote:

> > And thus I don't know why you can't just copy the minalloc
> > value to the maxalloc.

> Thats exactly what I did.
>
> But when you do that you also need to recalculate SS/SP, otherwise it most
> likely will point far beyond your program. Which is not a good idea.

If the SS/SP are disturbed it implies that the difference between
minalloc and maxalloc is put between the bss and stack.

In addition, it means when MSDOS is loading the program,
it would need to manipulate the SS itself, depending how
much memory between minalloc and maxalloc it was able
to allocate.

> But why don't you just try to make a program in which you enlarge the code,
> BSS and stack areas (several combinations) and look at how the fields in the
> EXE header change ? I found the EXE header info on the web and that is
> what I did to verify if I understood it right.

That's a good idea.

Thanks. Paul.

Muta (paul),

> If the SS/SP are disturbed it implies that the difference between
> minalloc and maxalloc is put between the bss and stack.

Nope. Minalloc refers to to the end of the combined the bss and following
stack segments. Maxalloc can go, as you read yourself, way beyond that.

But I owe you an apology : I remember updating SS/SP after setting MaxAlloc
to MinAlloc, but a peek into my program (still have it!) showed I made it
optional. In case I did not define a stack segment at all (but just kept
some BSS space free for it), causing SS:SP to be Zero in the EXE header.
And that ofcourse doesn't really work. So, I than can force it to be set to
the end of the BSS segment itself.

IOW, for a program in which a stack segment has been defined you only need,
as you mentioned, to copy MinAlloc into MaxAlloc.

I hope I did not create to much of a confusion.

> That's a good idea.

:-) Its the way I've had to, and still do figure out lots of stuff, which
sometimes makes you run into the darnest things. And even when full
documentation was/is available I always had/have the need to verify what was
said.

Regards,
Rudy Wieser

On Fri, 3 Dec 2021 09:45:00 +0100, Mateusz Viste wrote:
> 2021-12-03 at 00:38 -0800, muta...@gmail.com wrote:
>> So for all executables, all of memory is allocated until
>> released?
>
> AFAIK this is true only for COM files, where CS=DS=ES=SS.

EXE files can be made to allocate the maximum available memory. e.g.
compiling a tiny model program into an EXE.

On Fri, 3 Dec 2021 00:38:01 -0800 (PST), muta...@gmail.com wrote:
> On Friday, December 3, 2021 at 7:06:01 PM UTC+11, JJ wrote:
>
> So for all executables, all of memory is allocated until
> released?

For EXE files, it's header tells DOS how much memory should be allocated -
which can be specific amount, or maximum available.

> Do you think that ideally MSDOS 2.0 should have included
> a call to auto-release extra memory?

No, because that may break compatibility with old programs which are not
aware of MCBs. The program may end up corrupting the MCB following the
current one.

> PDOS/86 has the ability to add extra calls like that. But I
> need some way of testing whether I am running under a
> PDOS-compliant (or some other existing standard). I was
> thinking of getting the MSDOS version number and if it
> was 0.1 or something like that, then it meant additional
> calls were available to find more things like manufacturer
> and capability (extra INT 21H calls).
>
> BFN. Paul.

You can use the DOS OEM number returned by AH=30h. Though, that service is
DOS 2.0+ only. For DOS 1.0, I don't think there's a CP/M specification to
provide ID for CP/M variants.

JJ,

> EXE files can be made to allocate the maximum available memory.
> e.g. compiling a tiny model program into an EXE.

Hmmm... MaxAlloc getting set to $FFFF because of using the tiny model ? I
did not consider that - I always thought it was just a(n obnoxious) TLink
thing ...

Even after all these years I still learn new stuff. :-)

And I just remembered a problem I've never been able to figure out having to
do with not using the tiny model. Firing off a new subject in 3.. 2.. 1..

Regards,
Rudy Wieser

2021-12-04 at 14:00 +0700, JJ wrote:
> On Fri, 3 Dec 2021 09:45:00 +0100, Mateusz Viste wrote:
> > 2021-12-03 at 00:38 -0800, muta...@gmail.com wrote:
> >> So for all executables, all of memory is allocated until
> >> released?
> >
> > AFAIK this is true only for COM files, where CS=DS=ES=SS.
>
> EXE files can be made to allocate the maximum available memory. e.g.
> compiling a tiny model program into an EXE.

"can be made", yes, through fiddling with exe headers, but then that's
done on purpose, and not a by-design behavior, right?

Mateusz

On Saturday, December 4, 2021 at 6:01:23 PM UTC+11, JJ wrote:

> > PDOS/86 has the ability to add extra calls like that. But I
> > need some way of testing whether I am running under a
> > PDOS-compliant (or some other existing standard). I was
> > thinking of getting the MSDOS version number and if it
> > was 0.1 or something like that, then it meant additional
> > calls were available to find more things like manufacturer
> > and capability (extra INT 21H calls).

> You can use the DOS OEM number returned by AH=30h. Though, that service is
> DOS 2.0+ only. For DOS 1.0, I don't think there's a CP/M specification to
> provide ID for CP/M variants.

DOS 2.0+ is fine. Thanks for that.

I see that Freedos already has a number. I might make
contact with them. I really need a number that does not
represent a particular OEM but instead a "standard". And
I just realized that if MSDOS was theoretically updated to
follow the standard, they wouldn't be able to switch to that
new number. Hmmm. Maybe they could. Maybe everyone
who wished to follow the standard could switch to that
number, and then if they wanted the "true manufacturer"
they have to do another API call.

Any thoughts?

There are a few extensions I am after:

1. A memory allocation routine that takes a 32-bit number
of bytes and returns a far pointer. I might also have a flag
to say whether I want the memory to be allocated in the
first 1 MiB. This is for a theoretical processor where the
segment shift value can be changed from 4 to anything
up to 16, or the equivalent is being done using a selector
and GDT+LDT maxed out and lined up.

2. I want to know if there is a PDOS-generic style parameters
on the stack.

3. I want a routine that takes a far pointer and a 32-bit integer
and adds them together to produce a normalized far pointer
(huge pointer) as per current segment shift/selector rules.

4. I want a routine to get the command line.

I'm not 100% certain about this because I think I might have
to instead just create a new 8086 OS and not attempt to be
MSDOS-compatible.

BFN. Paul.

Mateusz,

> "can be made", yes, through fiddling with exe headers,
> but then that's done on purpose, and not a by-design
> behavior, right?

In the case of Borlands Tasm it seems to be by design. And with no option
to override *I* had to "fiddle" the EXE headers. :-\

And pardon me saying so, but how is "by design" *not* "on purpose" :-)

Regards,
Rudy Wieser

2021-12-04 at 13:00 +0100, R.Wieser wrote:
> In the case of Borlands Tasm it seems to be by design. And with no
> option to override *I* had to "fiddle" the EXE headers. :-\

Still, the Tasm tool does it on purpose. It has choice, it just decides
oterwise. When dealing with COM files, there is no choice.

> And pardon me saying so, but how is "by design" *not* "on purpose" :-)

The design of the COM-style loading led to simplistic stack
sizing and allocation. I doubt COM loading was designed with the
specific goal of allocating a full 64K of memory with stack at the end -
it was just a consequence of an earlier design choice.

In the case of EXE files, there are headers to deal with this
situation. A tool that sets the headers in a specific way does so on
purpose.

Mateusz

Mateusz,

> Still, the Tasm tool does it on purpose.

Thats how its normally done. Waiting for it to do it by accident could
take a loooong time. :-)

> It has choice, it just decides oterwise.

You sound as if you think it made the wrong one. Which decision do you
think should it have made and on what grounds ?

> When dealing with COM files, there is no choice.

Huh? I thought I was responding to a remark pertaining the EXE style
programs.

But yes, I know that with the COM format there isn't much choice in this
regard. As such I already removed it from consideration.

Regards,
Rudy Wieser

2021-12-04 at 15:55 +0100, R.Wieser wrote:
> You sound as if you think it made the wrong one. Which decision do
> you think should it have made and on what grounds ?

I don't think it made the wrong one. Or the good one. I do not discuss
the choice itself, surely it was backed up with careful consideration. I
only point out that it *is* a choice of the tool you use.

Now, just wondering aloud: is there an obvious reason why
Tasm does not pre-set MAXALLOC to be the same as MINALLOC? In the case
of high-level compilers, having a big MAXALLOC allows the runtime
to provide near pointers to the program via malloc(), but I assume that
Tasm, being an assembler, does not provide such conveniences.

Isn't it because the EXE header is, in fact, not generated by Tasm,
but by the linker (tlink)? Tlink might not know whether or not the
objects it links together come from Tasm or something else (that might
include libc calls). If that would be vaguely correct, then maybe
there's some tlink switch that could make it behave differently in this
context?

Mateusz

Mateusz,

> Now, just wondering aloud: is there an obvious reason why
> Tasm does not pre-set MAXALLOC to be the same as MINALLOC?
....
> Isn't it because the EXE header is, in fact, not generated
> by Tasm, but by the linker (tlink)?

Really ? You're suggesting that Tasm is some kind of culprit and than
"solve" it by telling us that its actually TLink who does it ? And with it
"answering" a question which was never asked ?

Nope, not going to play that game.

Goodbye.

Regards,
Rudy Wieser

2021-12-04 at 18:11 +0100, R.Wieser wrote:
> Really ? You're suggesting that Tasm is some kind of culprit

I never suggested that, you did. Almost exactly 6 hours ago.

> and than "solve" it by telling us that its actually TLink who does it

You seem to be misreading me. I am not solving anything, merely
suggesting that perhaps the problem is not where you are pointing at.

Mateusz

Mateusz,

> I never suggested that, you did. Almost exactly 6 hours ago.

Than you misunderstood. When I said "Borlands Tasm" I was referring to the
package. The Tasm as well as TLink programs, as you need both to be able to
create an executable.

> You seem to be misreading me. I am not solving anything,

Thats for sure.

> merely suggesting that perhaps the problem is not where
> you are pointing at.

And which "the problem" might that be ? I do not remember having a problem.
Just something that I like to have seen different.

Also, I don't think I pointed anywhere, as I do not even have a solid idea
what exactly causes the particular result.

.... Which I tried to make clear in that "almost exactly 6 hours ago" post.
Or did you really think that I only looked at one of the two involved
programs for such an option ?

Regards,
Rudy Wieser

On Sat, 4 Dec 2021 09:35:37 +0100, Mateusz Viste wrote:
> 2021-12-04 at 14:00 +0700, JJ wrote:
>> On Fri, 3 Dec 2021 09:45:00 +0100, Mateusz Viste wrote:
>>> 2021-12-03 at 00:38 -0800, muta...@gmail.com wrote:
>>>> So for all executables, all of memory is allocated until
>>>> released?
>>>
>>> AFAIK this is true only for COM files, where CS=DS=ES=SS.
>>
>> EXE files can be made to allocate the maximum available memory. e.g.
>> compiling a tiny model program into an EXE.
>
> "can be made", yes, through fiddling with exe headers, but then that's
> done on purpose, and not a by-design behavior, right?
>
> Mateusz

It's the default result of MASM as well as TASM. No command line switch
required. The EXE has to be made like that to make the binaries of source
code for old system not break the system.

On Sat, 4 Dec 2021 02:00:15 -0800 (PST), muta...@gmail.com wrote:
>
> I see that Freedos already has a number. I might make
> contact with them. I really need a number that does not
> represent a particular OEM but instead a "standard". And
> I just realized that if MSDOS was theoretically updated to
> follow the standard, they wouldn't be able to switch to that
> new number. Hmmm. Maybe they could. Maybe everyone
> who wished to follow the standard could switch to that
> number, and then if they wanted the "true manufacturer"
> they have to do another API call.

Nowadays, OS detection by detecting OS features is more reliable than simply
reading the OS ID.

On Sunday, December 5, 2021 at 2:30:58 AM UTC+11, Mateusz Viste wrote:

> In the case
> of high-level compilers, having a big MAXALLOC allows the runtime
> to provide near pointers to the program via malloc(),

Can you elaborate on this? I'm not familiar with how
other people implement their runtime libraries, but to
use a near pointer (offset only), the maxalloc won't
be able to be more than 64k, less the size of the stack,
less the size of the bss, less the size of the data section.

Thanks. Paul.