Message-ID:

An economist is a man who would marry Farrah Fawcett-Majors for her money.

devel / comp.lang.python / Re: Problem with concatenating two dataframes

Re: Problem with concatenating two dataframes

<mailman.185.1636228445.23718.python-list@python.org>

https://www.rocksolidbbs.com/devel/article-flat.php?id=20336&group=comp.lang.python#20336

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: python@mrabarnett.plus.com (MRAB)
Newsgroups: comp.lang.python
Subject: Re: Problem with concatenating two dataframes
Date: Sat, 6 Nov 2021 19:50:56 +0000
Lines: 93
Message-ID: <mailman.185.1636228445.23718.python-list@python.org>
References: <1670105892.779098.1636215375137.ref@mail.yahoo.com>
<1670105892.779098.1636215375137@mail.yahoo.com>
<737b7bd8-b718-2c4a-ee02-da077172bf4d@mrabarnett.plus.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Trace: news.uni-berlin.de +HW59sfCt8HzCeVOEjH/VAGcWrlztys5IYXr2bm9HX9A==
Return-Path: <python@mrabarnett.plus.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=plus.com header.i=@plus.com header.b=p4z0OjQJ;
dkim-adsp=none (unprotected policy); dkim-atps=neutral
X-Spam-Status: OK 0.002
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'argument': 0.04; 'error:':
0.05; 'e.g.': 0.07; 'that?': 0.07; '8bit%:62': 0.09;
'from:addr:python': 0.09; 'objects,': 0.09; 'pandas': 0.09;
'received:192.168.1.64': 0.09; 'skip:\xc2 20': 0.09;
'subject:two': 0.09; 'typeerror:': 0.09; 'append': 0.16;
'appended': 0.16; 'arguments': 0.16; 'batch': 0.16; 'column':
0.16; 'dataframe.': 0.16; 'dict': 0.16; 'dictionary.': 0.16;
'from:addr:mrabarnett.plus.com': 0.16; 'from:name:mrab': 0.16;
'iterable': 0.16; 'key-value': 0.16; 'mahmood': 0.16; 'message-
id:@mrabarnett.plus.com': 0.16; 'received:84.93': 0.16;
'received:84.93.230': 0.16; 'received:plus.net': 0.16; 'wrote:':
0.16; 'values': 0.17; 'to:addr:python-list': 0.20; 'list,': 0.24;
'object': 0.26; 'creating': 0.27; "isn't": 0.27; 'error': 0.29;
'header:User-Agent:1': 0.30; 'code,': 0.31; 'think': 0.32;
"doesn't": 0.32; 'python-list': 0.32; 'skip:7 10': 0.32;
'received:192.168.1': 0.32; 'but': 0.32; 'header:In-Reply-To:1':
0.34; 'trying': 0.35; 'following': 0.35; 'fix': 0.36;
'received:192.168': 0.37; 'file': 0.38; 'read': 0.38; 'wrote':
0.39; 'beginning': 0.40; 'otherwise,': 0.40; 'try': 0.40;
'should': 0.40; '8bit%:29': 0.60; 'skip:\xc2 10': 0.62; 'skip:i
20': 0.62; 'skip:b 20': 0.63; 'pass': 0.64; 'key': 0.64;
'definition': 0.64; 'items': 0.68; 'compare': 0.69; 'created.':
0.69; '8bit%:38': 0.70; '8bit%:31': 0.73; '8bit%:94': 0.75;
'8bit%:100': 0.76; 'supposed': 0.76; '8bit%:67': 0.81; '8bit%:97':
0.84; 'pairs': 0.84; 'skip:d 30': 0.86
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=plus.com; s=042019;
t=1636228258; bh=j7PqifvHYxckqKmeWTNUKgmO22ROMN/fAiW4d8KNIRI=;
h=Date:Subject:To:References:From:In-Reply-To;
b=p4z0OjQJ3rQOVfEBA4aYIy5HTRK6Jjk8AWGrl1NU1cd/u+0JVwJjv9CWc4Kd/70FD
gu7mTxmDxu8yVXWWB2zgar/85UqK2/5i9Rid9sQQhbtLavJJPcBVOEWxyEjRmHQ5jp
+QgMjfUtgUMPIJD0/PbsPMfF875F+Mg2/Mqkj5G976VIkalIHMrby2neQ8c4SlBldN
rR9lUcL/UiWdLEERHKwOeNI2r8/FQmSt3OUCDyI8pjA8yJmIBqa4C8mZkGLc+WZTsz
EDrPUHdLZrMjRH0a3SkLCZ7FgXqMFRisjD3rERcK6AOdT82ESfTn5z42ZAbOTJWpyk
9IAPMesoESAow==
X-Clacks-Overhead: "GNU Terry Pratchett"
X-CM-Score: 0.00
X-CNFS-Analysis: v=2.4 cv=HpkVp2fS c=1 sm=1 tr=0 ts=6186dca2
a=0nF1XD0wxitMEM03M9B4ZQ==:117 a=0nF1XD0wxitMEM03M9B4ZQ==:17
a=IkcTkHD0fZMA:10 a=Dbb6-eklgODtqPXLT_kA:9 a=QEXdDO2ut3YA:10
X-AUTH: mrabarnett@:2500
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
Thunderbird/91.3.0
Content-Language: en-GB
In-Reply-To: <1670105892.779098.1636215375137@mail.yahoo.com>
X-CMAE-Envelope: MS4xfA/mBvwC2pkK6WtagdpsRiGrGwQg3yGwvTwUnHvR3xfOhkuvPYn30JZlRzM46gdySoUK+1R9kKm+H5XdnEQLK5sf3RhzazL7Jv2SOKkt3QOPFSEtemAw
dcNrrkRkXxGueljwAwl79mGR5B1t/o7HVa5r4PZ+6nBKiG+kItugh6w98dxM0fSxzTawc1ExDIM304TYrLG9AGaEglsDygXSwGQ=
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.35
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <737b7bd8-b718-2c4a-ee02-da077172bf4d@mrabarnett.plus.com>
X-Mailman-Original-References: <1670105892.779098.1636215375137.ref@mail.yahoo.com>
<1670105892.779098.1636215375137@mail.yahoo.com>

by: MRAB - Sat, 6 Nov 2021 19:50 UTC

On 2021-11-06 16:16, Mahmood Naderan via Python-list wrote:
> In the following code, I am trying to create some key-value pairs in a dictionary where the first element is a name and the second element is a dataframe.
>
> # Creating a dictionary
> data = {'Value':[0,0,0]}
> kernel_df = pd.DataFrame(data, index=['M1','M2','M3'])
> dict = {'dummy':kernel_df}
> # dummy  ->          Value
> #               M1      0
> #               M2      0
> #               M3      0
>
>
> Then I read a file and create some batches and compare the name in the batch with the stored names in dictionary. If it doesn't exist, a new key-value (name and dataframe) is created. Otherwise, the Value column is appended to the existing dataframe.
>
>
> df = pd.read_csv('test.batch.csv')
> print(df)
> for i in range(0, len(df), 3):
>     print("\n------BATCH BEGIN")
>     batch_df = df.iloc[i:i+3]
>     name = batch_df.loc[i].at["Name"]
>     values = batch_df.loc[:,["Value"]]
>     print(name)
>     print(values)
>     print("------BATCH END")
>     if name in dict:
>         # Append values to the existing key
>         dict[name] = pd.concat( dict[name],values )   #### ERROR
>     else:
>         # Create a new pair in dictionary
>         dict[name] = values;
>
>
>
> As you can see in the output, the join statement has error.
>
>
>
>    ID Name Metric  Value
> 0   0   K1     M1     10
> 1   0   K1     M2      5
> 2   0   K1     M3     10
> 3   1   K2     M1     20
> 4   1   K2     M2     10
> 5   1   K2     M3     15
> 6   2   K1     M1      2
> 7   2   K1     M2      2
> 8   2   K1     M3      2
>
> ------BATCH BEGIN
> K1
>    Value
> 0     10
> 1      5
> 2     10
> ------BATCH END
>
> ------BATCH BEGIN
> K2
>    Value
> 3     20
> 4     10
> 5     15
> ------BATCH END
>
> ------BATCH BEGIN
> K1
>    Value
> 6      2
> 7      2
> 8      2
> ------BATCH END
>
>
>
>
> As it reaches the contact() statement, I get this error:
>
> TypeError: first argument must be an iterable of pandas objects, you passed an object of type "DataFrame"
>
>
> Based on the definition I wrote in the beginning of the code, "dict[name]" should be a dataframe. Isn't that?
>
> How can I fix that?
>
You're trying to concatenate by passing the 2 items as the first 2
arguments to pd.concat, but I think that you're supposed to pass them as
an _iterable_, e.g. a list, as the first argument to pd.concat.

Try this instead:

dict[name] = pd.concat([dict[name], values])

Subject	Author
Re: Problem with concatenating two dataframes	MRAB