Rocksolid Light

Welcome to RetroBBS

mail  files  register  newsreader  groups  login

Message-ID:  

MESSAGE ACKNOWLEDGED -- The Pershing II missiles have been launched.

devel / comp.lang.python / Re: ast.parse, ast.dump, but with comment preservation?

SubjectAuthor
* ast.parse, ast.dump, but with comment preservation?samue...@gmail.com
+- Re: ast.parse, ast.dump, but with comment preservation?Chris Angelico
+- Re: ast.parse, ast.dump, but with comment preservation?Barry
`* Re: ast.parse, ast.dump, but with comment preservation?lucas
 +- Re: ast.parse, ast.dump, but with comment preservation?samue...@gmail.com
 `- Re: ast.parse, ast.dump, but with comment preservation?samue...@gmail.com

1
ast.parse, ast.dump, but with comment preservation?

<f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=20830&group=comp.lang.python#20830

  copy link   Newsgroups: comp.lang.python
X-Received: by 2002:a37:bac2:: with SMTP id k185mr10981678qkf.685.1639625849633;
Wed, 15 Dec 2021 19:37:29 -0800 (PST)
X-Received: by 2002:a05:620a:254a:: with SMTP id s10mr11021752qko.654.1639625849356;
Wed, 15 Dec 2021 19:37:29 -0800 (PST)
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!news.misty.com!border2.nntp.dca1.giganews.com!nntp.giganews.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.lang.python
Date: Wed, 15 Dec 2021 19:37:29 -0800 (PST)
Injection-Info: google-groups.googlegroups.com; posting-host=50.234.189.45; posting-account=yMzBmAoAAAAoQ0hWTXoZ7y-jB_Aa_5gj
NNTP-Posting-Host: 50.234.189.45
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
Subject: ast.parse, ast.dump, but with comment preservation?
From: samuelmarks@gmail.com (samue...@gmail.com)
Injection-Date: Thu, 16 Dec 2021 03:37:29 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Lines: 23
 by: samue...@gmail.com - Thu, 16 Dec 2021 03:37 UTC

I wrote a little open-source tool to expose internal constructs in OpenAPI. Along the way, I added related functionality to:
- Generate/update a function prototype to/from a class
- JSON schema
- Automatically add type annotations to all function arguments, class attributes, declarations, and assignments

alongside a bunch of other features. All implemented using just the builtin modules (plus astor on Python < 3.9; and optionally black).

Now I'm almost at the point where I can run it—without issue—against, e.g., the entire TensorFlow codebase. Unfortunately this is causing huge `diff`s because the comments aren't preserved (and there are some whitespace issues… but I should be able to resolve the latter).

Is the only viable solution available to rewrite around redbaron | libcst? - I don't need to parse the comments just dump them out unedited whence they're found…

Thanks for any suggestions

PS: Library is https://github.com/SamuelMarks/cdd-python (might relicense with CC0… anyway too early for others to use; wait for the 0.1.0 release ;])

Re: ast.parse, ast.dump, but with comment preservation?

<mailman.5.1639637071.1567.python-list@python.org>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=20834&group=comp.lang.python#20834

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: rosuav@gmail.com (Chris Angelico)
Newsgroups: comp.lang.python
Subject: Re: ast.parse, ast.dump, but with comment preservation?
Date: Thu, 16 Dec 2021 17:44:19 +1100
Lines: 36
Message-ID: <mailman.5.1639637071.1567.python-list@python.org>
References: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
<CAPTjJmoCymrEzX+13q87MhHZ9WTe8Ps1LkM3wU3OBkVxTj4now@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Trace: news.uni-berlin.de P2Df41O0fkhlu70vw3Mp9wP1MjwZFuaWsXxpYVKzHr9g==
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=gmail.com header.i=@gmail.com header.b=OqC8V5Rw;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.002
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'url-ip:140.82.114/24':
0.03; 'url-ip:140.82/16': 0.03; 'comments': 0.03; 'bunch': 0.05;
'modules': 0.07; 'ast': 0.09; 'json': 0.09; 'open-source': 0.09;
'parse': 0.09; 'subject:but': 0.09; 'tensorflow': 0.09;
'url:github': 0.14; 'url-ip:140/8': 0.15; '2:47': 0.16;
'assignments': 0.16; 'attributes,': 0.16; 'builtin': 0.16;
'chrisa': 0.16; 'column': 0.16; 'expose': 0.16;
'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16;
'node': 0.16; 'rewrite': 0.16; 'wrote:': 0.16; 'python': 0.16;
'16,': 0.19; "aren't": 0.19; 'thu,': 0.19; 'to:addr:python-list':
0.20; 'code': 0.23; 'run': 0.23; '(and': 0.25; 'library': 0.26;
'function': 0.27; 'suggestions': 0.28; 'it,': 0.29; '8bit%:19':
0.31; 'dec': 0.31; 'do.': 0.32; 'resolve': 0.32; 'message-
id:@mail.gmail.com': 0.32; 'but': 0.32; "i'm": 0.33; 'there':
0.33; 'release': 0.34; 'able': 0.34; 'header:In-Reply-To:1': 0.34;
'received:google.com': 0.34; 'trying': 0.35;
'from:addr:gmail.com': 0.35; 'possibly': 0.36; 'source': 0.36;
'using': 0.37; "it's": 0.37; 'received:209.85': 0.37; 'class':
0.37; 'others': 0.37; 'could': 0.38; 'thanks': 0.38;
'received:209': 0.39; 'changes': 0.39; 'added': 0.39; 'text':
0.39; 'wrote': 0.39; 'want': 0.40; 'try': 0.40; 'should': 0.40;
'to:': 0.62; 'internal': 0.63; 'email addr:gmail.com': 0.63;
'full': 0.64; 'capable': 0.64; 'tool': 0.65; 'well': 0.65;
'entire': 0.67; 'ps:': 0.69; '2021': 0.71; 'little': 0.73;
'causing': 0.75; 'features.': 0.76; 'implemented': 0.76; 'need.':
0.84; 'prototype': 0.84; 'schema': 0.84; 'preserved': 0.91
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:content-transfer-encoding;
bh=/7rxbyjoOXBc4BqK+JWThSLPFyeomjybpYf/CBj0RVI=;
b=OqC8V5RwKOxvVpjeIVYyCKDdywBVTHbabD7apH+qJ9pE6vXgzT+bqxs2om4l9KUn+2
vZmtAWBMrt2qAva0BrOvRjtuU1/EdQVyI7VzmdjhUxMMPNNj7TLn0zVopSGN0fWV83Ka
kMJCcUlKyfIEKWW4wTX9pvnZiDlzWy5z3V4E2bM8vcEVleFE5v/AYvfoUhLRuUUaKM8O
ynk+EpJMM8OOX3agEqp+im7Tf9P9QYL6ZU+rBVUq0OTXcKx09k8W/zvxH4uodisq4GoB
Y5JrR3QcAM4PwFdwenAXRG3vjI2Uonv9LalpcYsWEcDd+lN9Av+48c6On9Hw68F4oWb8
iJkg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:content-transfer-encoding;
bh=/7rxbyjoOXBc4BqK+JWThSLPFyeomjybpYf/CBj0RVI=;
b=iomZoKfVMLlkWb1XGoc9RF+C5wpJic1ERzEfkb6nSWkx4/ewsaVyCr6pJVEDVnbX/F
jTJVy3ngX5ii2NNkXKbB7Ufpvd/VyrTIWqZDoTugsJSfyw4aNHF307ZF7Lli/wC30Ncv
gVUsFjY/Hi0c1xsJCzYa3Tm7wEFLFwSBHV4OhuXo7xyCr91FHuXXv1DJjXTQ4O0EJhcs
0/MkR/n7s79W1I2z9oUfysaiufEtChMkrzIpRPxxV+V4tj1O8+mIS24v9peAB+uKFd/K
BJzRbrjBGmGeK7kYIWOuebIj0Ktk2W9BRvQGzaPvvu/aENVOeCieunN+fR+S/7BFfk+i
7WkQ==
X-Gm-Message-State: AOAM533LwOKSsCY29zU7nql2Ia5OOnG3Z/Crrhekm3HrSHM8CkSMWqaP
qaJ55g0kBD18RSpCz0KPV2/UB0UflTq7fT4ci9VwO5qk
X-Google-Smtp-Source: ABdhPJwdk7yL/vNPkUG00aKY6by2IcfWz7UZnzN6tQVRz5CJMJqaeh/LuztkSz49jDXZme/a1yxY+c/vHkuDCbEeKnM=
X-Received: by 2002:adf:c605:: with SMTP id n5mr7446686wrg.564.1639637069712;
Wed, 15 Dec 2021 22:44:29 -0800 (PST)
In-Reply-To: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <CAPTjJmoCymrEzX+13q87MhHZ9WTe8Ps1LkM3wU3OBkVxTj4now@mail.gmail.com>
X-Mailman-Original-References: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
 by: Chris Angelico - Thu, 16 Dec 2021 06:44 UTC

On Thu, Dec 16, 2021 at 2:47 PM samue...@gmail.com
<samuelmarks@gmail.com> wrote:
>
> I wrote a little open-source tool to expose internal constructs in OpenAPI. Along the way, I added related functionality to:
> - Generate/update a function prototype to/from a class
> - JSON schema
> - Automatically add type annotations to all function arguments, class attributes, declarations, and assignments
>
> alongside a bunch of other features. All implemented using just the builtin modules (plus astor on Python < 3.9; and optionally black).
>
> Now I'm almost at the point where I can run it—without issue—against, e.g., the entire TensorFlow codebase. Unfortunately this is causing huge `diff`s because the comments aren't preserved (and there are some whitespace issues… but I should be able to resolve the latter).
>
> Is the only viable solution available to rewrite around redbaron | libcst? - I don't need to parse the comments just dump them out unedited whence they're found…
>
> Thanks for any suggestions
>
> PS: Library is https://github.com/SamuelMarks/cdd-python (might relicense with CC0… anyway too early for others to use; wait for the 0.1.0 release ;])

I haven't actually used it, but what you may want to try is lib2to3.
It's capable of full text reconstruction like you're trying to do.

Otherwise: Every AST node contains line and column information, so you
could possibly work the other way: keep the source code as well as the
AST, and make changes line by line as you have need.

ChrisA

Re: ast.parse, ast.dump, but with comment preservation?

<mailman.6.1639644151.1567.python-list@python.org>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=20835&group=comp.lang.python#20835

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: barry@barrys-emacs.org (Barry)
Newsgroups: comp.lang.python
Subject: Re: ast.parse, ast.dump, but with comment preservation?
Date: Thu, 16 Dec 2021 08:30:31 +0000
Lines: 39
Message-ID: <mailman.6.1639644151.1567.python-list@python.org>
References: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
<469ECCAD-6782-434D-B26E-37182E047369@barrys-emacs.org>
Mime-Version: 1.0 (1.0)
Content-Type: text/plain;
charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Trace: news.uni-berlin.de gluIDCdhA0hDy1vRCMVnqgC2oDt1dt7FaL7E1O1Cvwsg==
Return-Path: <barry@barrys-emacs.org>
X-Original-To: Python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=none reason="no signature";
dkim-adsp=none (unprotected policy); dkim-atps=neutral
X-Spam-Status: OK 0.000
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'url-ip:140.82/16': 0.03;
'comments': 0.03; 'bunch': 0.05; 'modules': 0.07; 'cc:addr:python-
list': 0.09; 'from:addr:barry': 0.09; 'json': 0.09; 'open-source':
0.09; 'parse': 0.09; 'received:217.70': 0.09;
'received:217.70.178': 0.09; 'received:gandi.net': 0.09;
'received:mail.gandi.net': 0.09; 'subject:but': 0.09;
'tensorflow': 0.09; 'cc:no real name:2**0': 0.14; 'url:github':
0.14; 'url:mailman': 0.15; 'url-ip:140/8': 0.15; '&lt;': 0.16;
'assignments': 0.16; 'attributes,': 0.16; 'barry': 0.16;
'builtin': 0.16; 'expose': 0.16; 'from:addr:barrys-emacs.org':
0.16; 'message-id:@barrys-emacs.org': 0.16; 'rewrite': 0.16;
'wrote:': 0.16; 'python': 0.16; 'uses': 0.19; "aren't": 0.19;
'libraries': 0.19; 'cc:addr:python.org': 0.20; 'code': 0.23;
'run': 0.23; '(and': 0.25; 'url-ip:188.166.95.178/32': 0.25; 'url-
ip:188.166.95/24': 0.25; 'url:listinfo': 0.25; 'cc:2**0': 0.25;
'url-ip:188.166/16': 0.25; 'library': 0.26; 'function': 0.27;
'suggestions': 0.28; '8bit%:19': 0.31; 'dec': 0.31; 'url-
ip:188/8': 0.31; 'think': 0.32; 'do.': 0.32; 'resolve': 0.32;
'but': 0.32; "i'm": 0.33; 'there': 0.33; 'release': 0.34; 'able':
0.34; 'header:In-Reply-To:1': 0.34; 'couple': 0.37; 'using': 0.37;
'class': 0.37; 'others': 0.37; 'thanks': 0.38; 'added': 0.39;
'wrote': 0.39; 'want': 0.40; 'should': 0.40; 'to:': 0.62;
'internal': 0.63; 'email addr:gmail.com': 0.63; 'tool': 0.65;
'look': 0.65; 'entire': 0.67; 'received:217': 0.67; 'ps:': 0.69;
'little': 0.73; 'causing': 0.75; 'features.': 0.76; 'implemented':
0.76; 'prototype': 0.84; 'schema': 0.84; 'preserved': 0.91
In-Reply-To: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
X-Mailer: iPad Mail (19B74)
X-Content-Filtered-By: Mailman/MimeDel 2.1.39
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <469ECCAD-6782-434D-B26E-37182E047369@barrys-emacs.org>
X-Mailman-Original-References: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
 by: Barry - Thu, 16 Dec 2021 08:30 UTC

> On 16 Dec 2021, at 03:49, samue...@gmail.com <samuelmarks@gmail.com> wrote:
>
> I wrote a little open-source tool to expose internal constructs in OpenAPI. Along the way, I added related functionality to:
> - Generate/update a function prototype to/from a class
> - JSON schema
> - Automatically add type annotations to all function arguments, class attributes, declarations, and assignments
>
> alongside a bunch of other features. All implemented using just the builtin modules (plus astor on Python < 3.9; and optionally black).
>
> Now I'm almost at the point where I can run it—without issue—against, e.g., the entire TensorFlow codebase. Unfortunately this is causing huge `diff`s because the comments aren't preserved (and there are some whitespace issues… but I should be able to resolve the latter).
>
> Is the only viable solution available to rewrite around redbaron | libcst? - I don't need to parse the comments just dump them out unedited whence they're found…
>
> Thanks for any suggestions

Have a look at the code that is used by https://github.com/asottile/pyupgrade
There are a couple of libraries that it uses that does what I think you want to do.

Barry

>
> PS: Library is https://github.com/SamuelMarks/cdd-python (might relicense with CC0… anyway too early for others to use; wait for the 0.1.0 release ;])
> --
> https://mail.python.org/mailman/listinfo/python-list

Re: ast.parse, ast.dump, but with comment preservation?

<mailman.7.1639652192.1567.python-list@python.org>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=20836&group=comp.lang.python#20836

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: lucas@bourneuf.net (lucas)
Newsgroups: comp.lang.python
Subject: Re: ast.parse, ast.dump, but with comment preservation?
Date: Thu, 16 Dec 2021 11:53:22 +0100
Lines: 27
Message-ID: <mailman.7.1639652192.1567.python-list@python.org>
References: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
<07e8e6c6-1dcb-f25e-1a7c-30b10db3d818@bourneuf.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Trace: news.uni-berlin.de i8T0OairKaebF5hxcjEcSgIZZ6bE4KQ6O0e/M7ryX84w==
Return-Path: <lucas@bourneuf.net>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="1024-bit key; unprotected key"
header.d=bourneuf.net header.i=@bourneuf.net header.b=hcd+xxM9;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.009
X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'url-ip:140.82/16': 0.03;
'comments': 0.03; 'bunch': 0.05; 'modules': 0.07; 'json': 0.09;
'open-source': 0.09; 'parse': 0.09; 'subject:but': 0.09;
'tensorflow': 0.09; 'url:github': 0.14; 'url-ip:140/8': 0.15;
'assignments': 0.16; 'attributes,': 0.16; 'builtin': 0.16;
'expose': 0.16; 'iirc,': 0.16; 'rewrite': 0.16; 'wrote:': 0.16;
'python': 0.16; "aren't": 0.19; 'to:addr:python-list': 0.20;
'maybe': 0.22; 'run': 0.23; '(and': 0.25; 'library': 0.26;
'function': 0.27; 'suggestions': 0.28; '8bit%:19': 0.31; 'code,':
0.31; 'aims': 0.32; 'empty': 0.32; 'resolve': 0.32; 'but': 0.32;
"i'm": 0.33; 'there': 0.33; 'release': 0.34; 'able': 0.34; 'same':
0.34; 'header:In-Reply-To:1': 0.34; 'source': 0.36; 'using': 0.37;
'class': 0.37; 'others': 0.37; 'thanks': 0.38; 'added': 0.39;
'wrote': 0.39; 'exact': 0.40; 'should': 0.40; 'including': 0.60;
'to:': 0.62; 'internal': 0.63; 'email addr:gmail.com': 0.63;
'tool': 0.65; 'entire': 0.67; 'ps:': 0.69; 'little': 0.73;
'causing': 0.75; 'features.': 0.76; 'implemented': 0.76;
'received:94': 0.76; 'header:Received:2': 0.84; 'prototype': 0.84;
'schema': 0.84; 'preserved': 0.91
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bourneuf.net;
s=default; t=1639651762;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:mime-version:mime-version:content-type:content-type:
content-transfer-encoding:content-transfer-encoding:
in-reply-to:in-reply-to:references:references;
bh=R3NVazojUaRYWXxv0d1AM2bbnsGk8EXviXbJBQqO7YQ=;
b=hcd+xxM9TR5DLRY8HxxNrS5wGxbHBjJPOSdxg1Q6AknCS4CpCuCznaQSjpBAXnz27MnU+r
PxooMwQOPmW6EuA47XlL7QGplrfSc1nNwsM/Y836TvXgO+04R1rix0KWelWz2pAQNfqZuD
S2J7qRtKX/wO4C1hi6L3RoAcSFnTgz8=
Content-Language: en-US
X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and
include these headers.
In-Reply-To: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
X-Migadu-Flow: FLOW_OUT
X-Migadu-Auth-User: bourneuf.net
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <07e8e6c6-1dcb-f25e-1a7c-30b10db3d818@bourneuf.net>
X-Mailman-Original-References: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
 by: lucas - Thu, 16 Dec 2021 10:53 UTC

Hi !

Maybe RedBaron may help you ?

https://github.com/PyCQA/redbaron

IIRC, it aims to conserve the exact same representation of the source
code, including comments and empty lines.

--lucas

On 16/12/2021 04:37, samue...@gmail.com wrote:
> I wrote a little open-source tool to expose internal constructs in OpenAPI. Along the way, I added related functionality to:
> - Generate/update a function prototype to/from a class
> - JSON schema
> - Automatically add type annotations to all function arguments, class attributes, declarations, and assignments
>
> alongside a bunch of other features. All implemented using just the builtin modules (plus astor on Python < 3.9; and optionally black).
>
> Now I'm almost at the point where I can run it—without issue—against, e.g., the entire TensorFlow codebase. Unfortunately this is causing huge `diff`s because the comments aren't preserved (and there are some whitespace issues… but I should be able to resolve the latter).
>
> Is the only viable solution available to rewrite around redbaron | libcst? - I don't need to parse the comments just dump them out unedited whence they're found…
>
> Thanks for any suggestions
>
> PS: Library is https://github.com/SamuelMarks/cdd-python (might relicense with CC0… anyway too early for others to use; wait for the 0.1.0 release ;])

Re: ast.parse, ast.dump, but with comment preservation?

<9eeae185-fe56-426f-95c6-62f8ac4e9922n@googlegroups.com>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=21067&group=comp.lang.python#21067

  copy link   Newsgroups: comp.lang.python
X-Received: by 2002:a05:6214:2583:: with SMTP id fq3mr2501245qvb.94.1641879040037;
Mon, 10 Jan 2022 21:30:40 -0800 (PST)
X-Received: by 2002:ad4:5bc9:: with SMTP id t9mr2550130qvt.70.1641879039773;
Mon, 10 Jan 2022 21:30:39 -0800 (PST)
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!news.misty.com!border2.nntp.dca1.giganews.com!nntp.giganews.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.lang.python
Date: Mon, 10 Jan 2022 21:30:39 -0800 (PST)
In-Reply-To: <mailman.7.1639652192.1567.python-list@python.org>
Injection-Info: google-groups.googlegroups.com; posting-host=50.234.189.45; posting-account=yMzBmAoAAAAoQ0hWTXoZ7y-jB_Aa_5gj
NNTP-Posting-Host: 50.234.189.45
References: <07e8e6c6-1dcb-f25e-1a7c-30b10db3d818@bourneuf.net>
<f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com> <mailman.7.1639652192.1567.python-list@python.org>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <9eeae185-fe56-426f-95c6-62f8ac4e9922n@googlegroups.com>
Subject: Re: ast.parse, ast.dump, but with comment preservation?
From: samuelmarks@gmail.com (samue...@gmail.com)
Injection-Date: Tue, 11 Jan 2022 05:30:40 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Lines: 47
 by: samue...@gmail.com - Tue, 11 Jan 2022 05:30 UTC

On Thursday, December 16, 2021 at 5:56:51 AM UTC-5, lucas wrote:
> Hi !
>
> Maybe RedBaron may help you ?
>
> https://github.com/PyCQA/redbaron
>
> IIRC, it aims to conserve the exact same representation of the source
> code, including comments and empty lines.
>
> --lucas
> On 16/12/2021 04:37, samue...@gmail.com wrote:
> > I wrote a little open-source tool to expose internal constructs in OpenAPI. Along the way, I added related functionality to:
> > - Generate/update a function prototype to/from a class
> > - JSON schema
> > - Automatically add type annotations to all function arguments, class attributes, declarations, and assignments
> >
> > alongside a bunch of other features. All implemented using just the builtin modules (plus astor on Python < 3.9; and optionally black).
> >
> > Now I'm almost at the point where I can run it—without issue—against, e.g., the entire TensorFlow codebase. Unfortunately this is causing huge `diff`s because the comments aren't preserved (and there are some whitespace issues… but I should be able to resolve the latter).
> >
> > Is the only viable solution available to rewrite around redbaron | libcst? - I don't need to parse the comments just dump them out unedited whence they're found…
> >
> > Thanks for any suggestions
> >
> > PS: Library is https://github.com/SamuelMarks/cdd-python (might relicense with CC0… anyway too early for others to use; wait for the 0.1.0 release ;])

Ended up writing my own CST and added it to that library of mine (link above).

My target is adding/removing/changing of: docstrings, function return types, function arguments, and Assign/AnnAssign. All but the last are now implemented.

I was careful not to replace code elsewhere in my codebase, so everything except my new CST code (in its own files) stays, and everything else works exclusively with the builtin `ast` module as before.

Re: ast.parse, ast.dump, but with comment preservation?

<e356bcdb-3907-4177-bb95-9f7c085ce72an@googlegroups.com>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=21074&group=comp.lang.python#21074

  copy link   Newsgroups: comp.lang.python
X-Received: by 2002:a05:620a:1988:: with SMTP id bm8mr3706106qkb.494.1641918302962;
Tue, 11 Jan 2022 08:25:02 -0800 (PST)
X-Received: by 2002:a05:622a:164b:: with SMTP id y11mr4341927qtj.87.1641918302694;
Tue, 11 Jan 2022 08:25:02 -0800 (PST)
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!news.misty.com!border2.nntp.dca1.giganews.com!nntp.giganews.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.lang.python
Date: Tue, 11 Jan 2022 08:25:02 -0800 (PST)
In-Reply-To: <mailman.7.1639652192.1567.python-list@python.org>
Injection-Info: google-groups.googlegroups.com; posting-host=50.234.189.45; posting-account=yMzBmAoAAAAoQ0hWTXoZ7y-jB_Aa_5gj
NNTP-Posting-Host: 50.234.189.45
References: <07e8e6c6-1dcb-f25e-1a7c-30b10db3d818@bourneuf.net>
<f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com> <mailman.7.1639652192.1567.python-list@python.org>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <e356bcdb-3907-4177-bb95-9f7c085ce72an@googlegroups.com>
Subject: Re: ast.parse, ast.dump, but with comment preservation?
From: samuelmarks@gmail.com (samue...@gmail.com)
Injection-Date: Tue, 11 Jan 2022 16:25:02 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Lines: 12
 by: samue...@gmail.com - Tue, 11 Jan 2022 16:25 UTC

> > PS: Library is https://github.com/SamuelMarks/cdd-python (might relicense with CC0… anyway too early for others to use; wait for the 0.1.0 release ;])
Ended up writing my own CST and added it to that library of mine (link above).

My target is adding/removing/changing of: docstrings, function return types, function arguments, and Assign/AnnAssign. All but the last are now implemented.

I was careful not to replace code elsewhere in my codebase, so everything except my new CST code (in its own files) stays, and everything else works exclusively with the builtin `ast` module as before.

devel / comp.lang.python / Re: ast.parse, ast.dump, but with comment preservation?

1
server_pubkey.txt

rocksolid light 0.9.81
clearnet tor