Rocksolid Light

Welcome to RetroBBS

mail  files  register  newsreader  groups  login

Message-ID:  

The only way to learn a new programming language is by writing programs in it. -- Brian Kernighan

devel / comp.lang.c++ / Re: too many open files (experimental/filesystem)

SubjectAuthor
* too many open files (experimental/filesystem)Jivanmukta
+- Re: too many open files (experimental/filesystem)Jivanmukta
+* Re: too many open files (experimental/filesystem)Paavo Helde
|`* Re: too many open files (experimental/filesystem)Jivanmukta
| `- Re: too many open files (experimental/filesystem)Jivanmukta
`* Re: too many open files (experimental/filesystem)Muttley
 `- Re: too many open files (experimental/filesystem)Jivanmukta

1
too many open files (experimental/filesystem)

<u7pq1n$17q2j$1@portraits.wsisiz.edu.pl>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=589&group=comp.lang.c%2B%2B#589

  copy link   Newsgroups: comp.lang.c++
Path: i2pn2.org!i2pn.org!news.chmurka.net!news.icm.edu.pl!wsisiz.edu.pl!.POSTED.2a01:110f:1532:2300:d727:9531:90de:445d!not-for-mail
From: jivanmukta@poczta.onet.pl (Jivanmukta)
Newsgroups: comp.lang.c++
Subject: too many open files (experimental/filesystem)
Date: Sat, 1 Jul 2023 20:07:51 +0200
Organization: http://www.wit.edu.pl
Message-ID: <u7pq1n$17q2j$1@portraits.wsisiz.edu.pl>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Sat, 1 Jul 2023 18:07:51 -0000 (UTC)
Injection-Info: portraits.wsisiz.edu.pl; posting-host="2a01:110f:1532:2300:d727:9531:90de:445d";
logging-data="1304659"; mail-complaints-to="abuse@wsisiz.edu.pl"
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.11.0
Content-Language: en-US
 by: Jivanmukta - Sat, 1 Jul 2023 18:07 UTC

In a function below I have an exception:

src/obfuscator.cpp (682), get_cmdline_options: get_filepaths:
Error during program execution: filesystem error: directory iterator
cannot open directory: Too many open files
[/home/robert/Projekty/kohana-cms/cms/cms/media/images/themes/clean/]

although in mentioned folder there is only one file.

#include <experimental/filesystem> //
http://en.cppreference.com/w/cpp/experimental/fs
namespace fs = std::experimental::filesystem;

wstrvector get_filepaths(fs::path path, string extensions, wstring
except_subdir_path, wstring prefix_dir) {
wstring except_subdir_path_no_prefix = except_subdir_path;
if (starts_with(except_subdir_path, prefix_dir)) {
except_subdir_path_no_prefix =
except_subdir_path.substr(prefix_dir.length());
}
string p = path.string();
p = wstr2str(normalize_path(str2wstr(p), wstr2str(dir_separator)[0]));
path = fs::path(p);
const fs::directory_iterator end {};
wstrvector filepaths;
if (exists(p)) {
if (ends_with(p, wstr2str(dir_separator + L".")) ||
ends_with(p, wstr2str(dir_separator + L"." + dir_separator))) {
for (fs::directory_iterator iter {path}; iter != end; ++iter) {
string e = iter->path().extension().string();
if (safe_substr(e, 0, 1) == ".") {
e = safe_substr(e, 1);
}
if (is_regular_file(iter->path()) && (extensions == ""
|| index_of_string(explode(",", extensions), e) >= 0)) {
filepaths.push_back(str2wstr(iter->path().string()));
} else if (is_directory(iter->path()) &&
(except_subdir_path_no_prefix == L"" ||
str2wstr(iter->path().string()).find(except_subdir_path_no_prefix) ==
wstring::npos)) {
wstrvector subfiles = get_filepaths(iter->path(),
extensions, except_subdir_path, prefix_dir); // recursion
filepaths.insert(filepaths.end(), subfiles.begin(),
subfiles.end());
}
}
} else {
for (fs::directory_iterator iter {path}; iter != end; ++iter) {
string e = iter->path().extension().string();
if (safe_substr(e, 0, 1) == ".") {
e = safe_substr(e, 1);
}
if (is_regular_file(iter->path()) && (extensions == ""
|| index_of_string(explode(",", extensions), e) >= 0)) {
filepaths.push_back(str2wstr(iter->path().string()));
} else if (is_directory(iter->path()) &&
(except_subdir_path_no_prefix == L"" ||
str2wstr(iter->path().string()).find(except_subdir_path_no_prefix) ==
wstring::npos)) {
wstrvector subfiles = get_filepaths(iter->path(),
extensions, except_subdir_path, prefix_dir); // recursion
filepaths.insert(filepaths.end(), subfiles.begin(),
subfiles.end());
}
}
}
}
return filepaths;
}

Re: too many open files (experimental/filesystem)

<u7qo8o$18p1b$1@portraits.wsisiz.edu.pl>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=592&group=comp.lang.c%2B%2B#592

  copy link   Newsgroups: comp.lang.c++
Path: i2pn2.org!i2pn.org!news.chmurka.net!news.icm.edu.pl!wsisiz.edu.pl!.POSTED.2a01:110f:1532:2300:bed5:30df:267b:4f9f!not-for-mail
From: jivanmukta@poczta.onet.pl (Jivanmukta)
Newsgroups: comp.lang.c++
Subject: Re: too many open files (experimental/filesystem)
Date: Sun, 2 Jul 2023 04:43:36 +0200
Organization: http://www.wit.edu.pl
Message-ID: <u7qo8o$18p1b$1@portraits.wsisiz.edu.pl>
References: <u7pq1n$17q2j$1@portraits.wsisiz.edu.pl>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Sun, 2 Jul 2023 02:43:36 -0000 (UTC)
Injection-Info: portraits.wsisiz.edu.pl; posting-host="2a01:110f:1532:2300:bed5:30df:267b:4f9f";
logging-data="1336363"; mail-complaints-to="abuse@wsisiz.edu.pl"
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.11.0
Content-Language: en-US
In-Reply-To: <u7pq1n$17q2j$1@portraits.wsisiz.edu.pl>
 by: Jivanmukta - Sun, 2 Jul 2023 02:43 UTC

I change function's code. Still problem.

wstrvector get_filepaths(fs::path path, string extensions, wstring
except_subdir_path, wstring prefix_dir) {
if (path.string().find("vendor") != string::npos &&
except_subdir_path.find(L"kohana-multi-site") != string::npos) {
path = path;
} wstring except_subdir_path_no_prefix = except_subdir_path;
if (starts_with(except_subdir_path, prefix_dir)) {
except_subdir_path_no_prefix =
except_subdir_path.substr(prefix_dir.length());
}
string p = path.string();
p = wstr2str(normalize_path(str2wstr(p), wstr2str(dir_separator)[0]));
path = fs::path(p);
wstrvector filepaths;
if (exists(p)) {
/*
if (ends_with(p, wstr2str(dir_separator + L".")) ||
ends_with(p, wstr2str(dir_separator + L"." + dir_separator))) {
*/
const fs::directory_iterator end {};
for (fs::directory_iterator iter {path}; iter != end; ++iter) {
string e = iter->path().extension().string();
if (safe_substr(e, 0, 1) == ".") {
e = safe_substr(e, 1);
}
if (is_regular_file(iter->path()) && (extensions == "" ||
index_of_string(explode(",", extensions), e) >= 0)) {
filepaths.push_back(str2wstr(iter->path().string()));
} else if (is_directory(iter->path()) &&
(except_subdir_path_no_prefix == L"" ||
str2wstr(iter->path().string()).find(except_subdir_path_no_prefix) ==
wstring::npos)) {
wstrvector subfiles = get_filepaths(iter->path(),
extensions, except_subdir_path, prefix_dir); // recursion
filepaths.insert(filepaths.end(), subfiles.begin(),
subfiles.end());
}
}
}
return filepaths;
}

Re: too many open files (experimental/filesystem)

<u7r7vt$38k97$1@dont-email.me>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=593&group=comp.lang.c%2B%2B#593

  copy link   Newsgroups: comp.lang.c++
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: eesnimi@osa.pri.ee (Paavo Helde)
Newsgroups: comp.lang.c++
Subject: Re: too many open files (experimental/filesystem)
Date: Sun, 2 Jul 2023 10:11:56 +0300
Organization: A noiseless patient Spider
Lines: 17
Message-ID: <u7r7vt$38k97$1@dont-email.me>
References: <u7pq1n$17q2j$1@portraits.wsisiz.edu.pl>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Sun, 2 Jul 2023 07:11:57 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="cb49e6c4cc4b60b7d2595b0086ac322c";
logging-data="3428647"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+rD6DSmDcAQ0aW+dR5biWPhc8yr3hvnv4="
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
Thunderbird/102.12.0
Cancel-Lock: sha1:+/3ML9nDftA5m+PdPJsLyV6HIoQ=
In-Reply-To: <u7pq1n$17q2j$1@portraits.wsisiz.edu.pl>
Content-Language: en-US
 by: Paavo Helde - Sun, 2 Jul 2023 07:11 UTC

01.07.2023 21:07 Jivanmukta kirjutas:
> In a function below I have an exception:
>
> src/obfuscator.cpp (682), get_cmdline_options: get_filepaths:
> Error during program execution: filesystem error: directory iterator
> cannot open directory: Too many open files
> [/home/robert/Projekty/kohana-cms/cms/cms/media/images/themes/clean/]
>
> although in mentioned folder there is only one file.
>

You are hitting the file descriptor limit in a single process.

As your function is recursive, the most obvious suspect is infinite
recursion.

Re: too many open files (experimental/filesystem)

<u7ran0$38sus$1@dont-email.me>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=595&group=comp.lang.c%2B%2B#595

  copy link   Newsgroups: comp.lang.c++
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: Muttley@dastardlyhq.com
Newsgroups: comp.lang.c++
Subject: Re: too many open files (experimental/filesystem)
Date: Sun, 2 Jul 2023 07:58:24 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 7
Message-ID: <u7ran0$38sus$1@dont-email.me>
References: <u7pq1n$17q2j$1@portraits.wsisiz.edu.pl>
Injection-Date: Sun, 2 Jul 2023 07:58:24 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="71f235578ec82bbbceceebcd8d4357c1";
logging-data="3437532"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+IjqSGcD5BdAVLQPGqxx6N"
Cancel-Lock: sha1:oICt6mSDUG2GqnYV+2g2hRc7TiI=
 by: Muttley@dastardlyhq.com - Sun, 2 Jul 2023 07:58 UTC

On Sat, 1 Jul 2023 20:07:51 +0200
Jivanmukta <jivanmukta@poczta.onet.pl> wrote:
>In a function below I have an exception:

Well good luck! There are better things to do in life than wade through
that mess. Pay me and I'll do it.

Re: too many open files (experimental/filesystem)

<u7uuas$1f7d0$1@portraits.wsisiz.edu.pl>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=621&group=comp.lang.c%2B%2B#621

  copy link   Newsgroups: comp.lang.c++
Path: i2pn2.org!i2pn.org!news.chmurka.net!news.icm.edu.pl!wsisiz.edu.pl!.POSTED.2a01:110f:1532:2300:9417:a1ca:5195:5056!not-for-mail
From: jivanmukta@poczta.onet.pl (Jivanmukta)
Newsgroups: comp.lang.c++
Subject: Re: too many open files (experimental/filesystem)
Date: Mon, 3 Jul 2023 18:51:40 +0200
Organization: http://www.wit.edu.pl
Message-ID: <u7uuas$1f7d0$1@portraits.wsisiz.edu.pl>
References: <u7pq1n$17q2j$1@portraits.wsisiz.edu.pl>
<u7r7vt$38k97$1@dont-email.me>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Mon, 3 Jul 2023 16:51:41 -0000 (UTC)
Injection-Info: portraits.wsisiz.edu.pl; posting-host="2a01:110f:1532:2300:9417:a1ca:5195:5056";
logging-data="1547680"; mail-complaints-to="abuse@wsisiz.edu.pl"
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.11.0
Content-Language: en-US
In-Reply-To: <u7r7vt$38k97$1@dont-email.me>
 by: Jivanmukta - Mon, 3 Jul 2023 16:51 UTC

W dniu 2.07.2023 o 09:11, Paavo Helde pisze:
> 01.07.2023 21:07 Jivanmukta kirjutas:
>> In a function below I have an exception:
>>
>> src/obfuscator.cpp (682), get_cmdline_options: get_filepaths:
>> Error during program execution: filesystem error: directory iterator
>> cannot open directory: Too many open files
>> [/home/robert/Projekty/kohana-cms/cms/cms/media/images/themes/clean/]
>>
>> although in mentioned folder there is only one file.
>>
>
> You are hitting the file descriptor limit in a single process.
>
> As your function is recursive, the most obvious suspect is infinite
> recursion.
>
>

I solved the problem by adding
&& (except_subdir_path_no_prefix == L"" ||
str2wstr(iter->path().string()).find(except_subdir_path_no_prefix) ==
wstring::npos)
to the IF is_regular_file and by using fs::recursive_directory_iterator.
There was no infinite recursion but tested direcotries structure was
very large.

wstrvector get_filepaths(fs::path path, string extensions, wstring
except_subdir_path, wstring prefix_dir) {
if (path.string().find("vendor") != string::npos &&
except_subdir_path.find(L"kohana-multi-site") != string::npos) {
path = path;
} wstring except_subdir_path_no_prefix = except_subdir_path;
if (starts_with(except_subdir_path, prefix_dir)) {
except_subdir_path_no_prefix =
except_subdir_path.substr(prefix_dir.length());
}
string p = path.string();
p = wstr2str(normalize_path(str2wstr(p), wstr2str(dir_separator)[0]));
path = fs::path(p);
wstrvector filepaths;
if (exists(p)) {
/*
if (ends_with(p, wstr2str(dir_separator + L".")) ||
ends_with(p, wstr2str(dir_separator + L"." + dir_separator))) {
*/
const fs::directory_iterator end {};
for (fs::directory_iterator iter {path}; iter != end; ++iter) {
string e = iter->path().extension().string();
if (safe_substr(e, 0, 1) == ".") {
e = safe_substr(e, 1);
}
if (is_regular_file(iter->path()) && (extensions == "" ||
index_of_string(explode(",", extensions), e) >= 0)) {
filepaths.push_back(str2wstr(iter->path().string()));
} else if (is_directory(iter->path()) &&
(except_subdir_path_no_prefix == L"" ||
str2wstr(iter->path().string()).find(except_subdir_path_no_prefix) ==
wstring::npos)) {
wstrvector subfiles = get_filepaths(iter->path(),
extensions, except_subdir_path, prefix_dir); // recursion
filepaths.insert(filepaths.end(), subfiles.begin(),
subfiles.end());
}
}
}
return filepaths;
}

Re: too many open files (experimental/filesystem)

<u8584v$1lt08$1@portraits.wsisiz.edu.pl>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=675&group=comp.lang.c%2B%2B#675

  copy link   Newsgroups: comp.lang.c++
Path: i2pn2.org!i2pn.org!news.chmurka.net!news.icm.edu.pl!wsisiz.edu.pl!.POSTED.2a01:110f:1532:2300:d21b:2853:4afc:ac14!not-for-mail
From: jivanmukta@poczta.onet.pl (Jivanmukta)
Newsgroups: comp.lang.c++
Subject: Re: too many open files (experimental/filesystem)
Date: Thu, 6 Jul 2023 04:15:59 +0200
Organization: http://www.wit.edu.pl
Message-ID: <u8584v$1lt08$1@portraits.wsisiz.edu.pl>
References: <u7pq1n$17q2j$1@portraits.wsisiz.edu.pl>
<u7ran0$38sus$1@dont-email.me>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 6 Jul 2023 02:15:59 -0000 (UTC)
Injection-Info: portraits.wsisiz.edu.pl; posting-host="2a01:110f:1532:2300:d21b:2853:4afc:ac14";
logging-data="1766408"; mail-complaints-to="abuse@wsisiz.edu.pl"
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.11.0
Content-Language: en-US
In-Reply-To: <u7ran0$38sus$1@dont-email.me>
 by: Jivanmukta - Thu, 6 Jul 2023 02:15 UTC

W dniu 2.07.2023 o 09:58, Muttley@dastardlyhq.com pisze:
> On Sat, 1 Jul 2023 20:07:51 +0200
> Jivanmukta <jivanmukta@poczta.onet.pl> wrote:
>> In a function below I have an exception:
>
> Well good luck! There are better things to do in life than wade through
> that mess. Pay me and I'll do it.
>
Do better things.
I will not pay you.

Re: too many open files (experimental/filesystem)

<u85h9l$1m6ov$1@portraits.wsisiz.edu.pl>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=677&group=comp.lang.c%2B%2B#677

  copy link   Newsgroups: comp.lang.c++
Path: i2pn2.org!i2pn.org!news.chmurka.net!news.icm.edu.pl!wsisiz.edu.pl!.POSTED.2a01:110f:1532:2300:c7e6:22f3:2fee:596d!not-for-mail
From: jivanmukta@poczta.onet.pl (Jivanmukta)
Newsgroups: comp.lang.c++
Subject: Re: too many open files (experimental/filesystem)
Date: Thu, 6 Jul 2023 06:52:05 +0200
Organization: http://www.wit.edu.pl
Message-ID: <u85h9l$1m6ov$1@portraits.wsisiz.edu.pl>
References: <u7pq1n$17q2j$1@portraits.wsisiz.edu.pl>
<u7r7vt$38k97$1@dont-email.me> <u7uuas$1f7d0$1@portraits.wsisiz.edu.pl>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Thu, 6 Jul 2023 04:52:05 -0000 (UTC)
Injection-Info: portraits.wsisiz.edu.pl; posting-host="2a01:110f:1532:2300:c7e6:22f3:2fee:596d";
logging-data="1776415"; mail-complaints-to="abuse@wsisiz.edu.pl"
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.11.0
Content-Language: en-US
In-Reply-To: <u7uuas$1f7d0$1@portraits.wsisiz.edu.pl>
 by: Jivanmukta - Thu, 6 Jul 2023 04:52 UTC

Sorry.

wstrvector get_filepaths(fs::path path, string extensions, wstring
except_subdir_path, wstring prefix_dir) {
wstring except_subdir_path_no_prefix = except_subdir_path;
if (starts_with(except_subdir_path, prefix_dir)) {
except_subdir_path_no_prefix =
except_subdir_path.substr(prefix_dir.length());
}
TRACE("except_subdir_path_no_prefix: " <<
except_subdir_path_no_prefix);
string p = path.string();
p = wstr2str(normalize_path(str2wstr(p), wstr2str(dir_separator)[0]));
path = fs::path(p);
wstrvector filepaths;
if (exists(p)) {
for (auto f : fs::recursive_directory_iterator(path)) {
string e = fs::path(f).extension().string();
if (safe_substr(e, 0, 1) == ".") {
e = safe_substr(e, 1);
}
if (is_regular_file(f) && (extensions == "" ||
index_of_string(explode(",", extensions), e) >= 0)
&& (except_subdir_path_no_prefix == L"" ||
str2wstr(fs::path(f).string()).find(except_subdir_path_no_prefix) ==
wstring::npos)) {
filepaths.push_back(str2wstr(fs::path(f).string()));
} else if (is_directory(f) && (except_subdir_path_no_prefix
== L"" ||
str2wstr(fs::path(f).string()).find(except_subdir_path_no_prefix) ==
wstring::npos)) {
filepaths.push_back(str2wstr(fs::path(f).string()));
}
}
}
return filepaths;
}

1
server_pubkey.txt

rocksolid light 0.9.8
clearnet tor