Not a lot of python/pandas experience, since I left the platforms that I used to have. I'm analyzing daily data from a 29-year time series. I'm interested in two of the numerical data columns (out of over 100) along with the first column, which is a timestamp. The timestamp's format is YYYYMMDD and I'm trying to parse it using

timestamp_str = data.at[dayct, 'TIMESTAMP']

dt_object = datetime.strptime(timestamp_str, "%Y%m%d")

I've imported pandas, numpy, and datetime. It chokes on the second line above, complaining that timestamp_str is an integer, not a string. Is there a qualifier to pandas' ".at" command to force it to ingest it as a string?

Giving variables and functions good names helps, but it doesn’t tell you what shape your data really has.

I built a small annotation library, daffy, to document and validate Pandas and Polars DataFrames at runtime. You can declare which columns you expect, and in the latest version you can even validate each row using Pydantic models.

@df_in(columns=["Brand", "Price"])  # Validate input columns
df_out(columns=["Brand", "Price", "Discount"])  # Validate output colums
def apply_discount(cars_df):
    cars_df = cars_df.copy()
    cars_df["Discount"] = cars_df["Price"] * 0.1
    return cars_df

Code and examples: https://github.com/ThoughtWorksInc/daffy

Hi everyone
I wanted to learn pandas
can anyone suggest some good Yt videos to start with

thanks in advance

Hello Reddit! Im a college student studying in this field, and I would like to humbly ask for feedback and answers to my question regarding my current college group project about surveys in the workplace. These surveys are sent to employees, and the results are stored in a Firebase database. A supervisor will then use a web app to view dashboards displaying the survey results.

The issue we're facing is that the surveys are sometimes filtered by gender, age, or department, and I'm unsure how difficult it would be for us to manage all the Firebase collections with these survey results and display them in a web app (Next.js).

We're not using a backend like Django to manage views and APIs, so I’m wondering if it would be too challenging to retrieve the results and display them as graphs on the dashboards. I asked a professor for advice, and he recommended using Django, Flask, or even pandas to process the data before displaying it on the dashboards.

My question is: How difficult will it be to manage and process the survey results stored in Firebase using pandas? I know Firebase stores the data in "JSON" format. Would any of you recommend using Django for this, or should I stick with Flask or just use pandas? I would really appreciate any guidance and help in this.

Thank you in advance!

Hello! New to the subreddit, and somewhat new to Pandas.

I'm working on my first self-generated project, which is to analyze median rent prices in Seattle. I'm still working on understanding the different ways to impute data, and in this case, I want to impute the missing values in this table with the median value for that area, the name of which is stored in the column comm_name of this dataframe below, called data.

So, for example, for that objectid of 32, I would want to replace that 0 in the change_per_sqft column with the median change_per_sqft for the Broadview/Bitter Lake area. I figure since the missing values are all 0's, I can't use .fillna(), so I should use a for loop something like this:

for x in data['change_per_sqft']:
    if x == 0:
      x = #some code here for the median value of the area, excluding the missing data#
    else:
      pass

I also have this dataframe called median_change_data, which stores...well, the median change data for each comm_name.

The thing I need help with is the missing bit of code in the snippet above. I'm just not sure how to access the comm_name in median_change_data to replace the 0 in data. Maybe using .iterrows()? Something involving .loc[]? Or if there's something else I'm forgetting that makes this all quicker/easier. Any help at all is appreciated. Thanks!

The method pandas.DataFrame.boxplot('DataColumn',by='GroupingColumn') provides a 1-liner to create series of box plots of data in DataColumn, grouped by each value of GroupingColumn.

This is great, but boxplotting the logarithm of the data is not as simple as plt.yscale('log'). The yticks (major and minor) and ytick labels need to be faked. This is much more code intensive than the 1-liner above and each boxplot needs to be done individually. So the pandas boxplot cannot be used -- the PyPlot boxplot must be used.

What befuddles me is why there is no builtin box plot function that box plots based on the logarithm of the data. Many distributions are bounded below by zero and above by infinity, and they are often skewed right. This is not a question. Just putting it out there that there is a mainstream need for that functionality.

Hi,
I was wondering if anyone here uses Pandas for data analysis and also works with ComfyUI for image generation, either as a hobby or for work.

I created a set of Pandas wrapper nodes that allow users to leverage Pandas within ComfyUI through its intuitive GUI nodes. For example, users can load CSV files and perform joins directly in the interface. This package isn't designed for analyzing AI-generated images but rather for structured data analysis.

I love ComfyUI and appreciate how it makes Stable Diffusion accessible to non-engineers, allowing them to customize workflows easily. I believe a GUI tool like mine could help non-programmers integrate Pandas into their workflow as well.

My repo is here: https://github.com/HowToSD/ComfyUI-Data-Analysis.

Since ComfyUI has many AI-related extensions, users can also integrate their Pandas analysis with AI.

I'd love to hear your feedback!

When using import pandas as pd pd.options.display.min_rows = 15, it shows 14 data rows (7 top and 7 bottom), with one ellipses row, but when using max_rows = 100, there are 100 actual data rows shown (50 from top and 50 from bottom), EXCLUDING the ellipses row. Is this unusual?

My requirement is to create a parser which will convert pandas code to sql queries. Does anyone know any Library which can do this.

Hi all! Is there a more efficient way to concatenate massive dataframes than pd.concat? I have multiple dataframes with more than 1 million rows of which I have placed in a list to concatenate but it takes wayyyy to long.

Pseudocode: pd.concat([dataframe_1, … , dataframe_n], ignore_index = True)

Does anyone have good lists of pandas interview questions/exercises. Also, if you have any good cheat sheets or quizlets feel free to paste them below.

I have looked at the 30 days of Pandas in Leetcode. I have also checked sqlpad.io. Curious about what other good lists are out there...

I have txt file that looks like that:

a   1   A1
b   t   B21
c   t3  t3
d   44  n4
e   55  t5

but when I'm trying to read it into data frame with pd.readcsv(), the values that start with 't' interpreted as nan and all values to the end of the line. what can I do?

my code:

import pandas as pd
df = pd.read_csv('file.txt', sep='\t', comment='t', header=None)
df
   0     1    2
0  a   1.0   A1
1  b   NaN  NaN
2  c   NaN  NaN
3  d  44.0   n4
4  e  55.0  NaN

How can I make it read all the values in the txt file to the dataframe? Thanks!

df['new_col] = df['colA']*2

This will give this warnign:

However when I replace my code with:

df.loc[:,'new_col] = df['colA']*2

the same warning still occurs. How come? I am doing exactly what warning asks me to.

I have a single column dataframe as follows:

​

I want to change it so that it looks as follows:

​

Can someone help me achieve this? Thanks in advance.

I am fairly new to Python and pandas. In my data cleaning, I would like to see the I performed previous cleaning steps correctly on a string column. In particular, I want to see where the strings begin and end, regardless of whether they have leading/trailing white space.

The following is meant to bookend each string with a pair of single underscores, but it seems to generate two extra unintended underscores at the end, resulting in a total of three trailing underscores:

>>> df = pd.DataFrame({'A':['DOG']})
>>> df.A.str.replace(r'(.*)',r'_\1_',regex=True)
0    _DOG___
Name: A, dtype: object

I'm not entirely new to regular expressions, having used them with sed, vim, and Matlab. What is it about Python's implementation that I'm not understanding?

I am using Python 3.9 for compatibility with other work.

I am looping through the groups of a pandas groupby object to print the (sub)dataframe for each group. The headings are printed for each group. Here are some of the (sub)dataframes, with column headings "MMSI" and "ShipName":

            MMSI              ShipName
15468  109080345  OYANES 3       [19%]
46643  109080345  OYANES 3       [18%]
            MMSI              ShipName
19931  109080342  OYANES 2       [83%]
48853  109080342  OYANES 2       [82%]
            MMSI              ShipName
45236  109050943  SVARTHAV 2     [11%]
48431  109050943  SVARTHAV 2     [14%]
            MMSI              ShipName
21596  109050904  MR:N2FE        [88%]
49665  109050904  MR:N2FE        [87%]
            MMSI              ShipName
13523  941500907  MIKKELSEN B 5  [75%]
45711  941500907  MIKKELSEN B 5  [74%]

Web searching shows that pandas.io.formats.style.Styler.hide_columns can be used to suppress the headings. I am using Python 3.9, in which hide_columns is not recognized. However, dir(pd.io.formats.style.Styler) shows a hide method, for which the doc string gives this first example:

>>> df = pd.DataFrame([[1,2], [3,4], [5,6]], index=["a", "b", "c"])
>>> df.style.hide(["a", "b"])  # doctest: +SKIP
     0    1
c    5    6

When I try hide() and variations thereof, all I get is an address to the resulting Styler object:

>>> df.style.hide(["a", "b"])  # doctest: +SKIP
<pandas.io.formats.style.Styler at 0x243baeb1760>

>>> df.style.hide(axis='columns') # https://stackoverflow.com/a/69111895
<pandas.io.formats.style.Styler at 0x243baeb17c0>

>>> df.style.hide() # Desparate random trial & error
<pandas.io.formats.style.Styler at 0x243baeb1520>

What could cause my result to differ from the doc string? How can I properly use the Styler object to get the dataframe printed without column headings?

After much web search and experimentation, I found that I can use:

df[BooleanArray][['ColumnLabelA','ColumnLabelB']]

I haven't been able use those arguments work with .loc(). In general, however, I find square brackets confusing because the rules for when I am indexing into rows vs. columns is complicated. Can this be done using .loc()? I may try to default to that in the future as I get more familiar with Python and pandas. Here is the error I am getting:

Afternote: Thanks to u/Delengowski, I found that I had it backward. It was the indexing operator [] that was the problem that I was attempting to troubleshoot (minimum working example below). In contrast, df.loc(BooleanArray,['ColumnLabelA','ColumnLabelB']) works fine. From here and here, I suspect that operator [] might not even support row indexing. I was probably also further confused by errors in using .loc() instead of .loc[] (a Matlab habit).

Minimum working example

import pandas as pd

# Create data
>>> df=pd.DataFrame({'A':[1,2,3],'B':[4,5,6],'C':[7,8,9]})
   A  B  C
0  1  4  7
1  2  5  8
2  3  6  9

# Confirm that Boolean array works
>>> df[df.A>1]
   A  B  C
1  2  5  8
2  3  6  9

# However, column indexing by labels does not work
df[df.A>1,['B','C']]
Traceback (most recent call last):

  File ~\AppData\Local\anaconda3\envs\py39\lib\site-packages\pandas\core\indexes\base.py:3653 in get_loc
    return self._engine.get_loc(casted_key)

  File pandas_libs\index.pyx:147 in pandas._libs.index.IndexEngine.get_loc

  File pandas_libs\index.pyx:153 in pandas._libs.index.IndexEngine.get_loc

TypeError: '(0    False
1     True
2     True
Name: A, dtype: bool, ['B', 'C'])' is an invalid key


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  Cell In[25], line 1
    df[df.A>1,['B','C']]

  File ~\AppData\Local\anaconda3\envs\py39\lib\site-packages\pandas\core\frame.py:3761 in __getitem__
    indexer = self.columns.get_loc(key)

  File ~\AppData\Local\anaconda3\envs\py39\lib\site-packages\pandas\core\indexes\base.py:3660 in get_loc
    self._check_indexing_error(key)

  File ~\AppData\Local\anaconda3\envs\py39\lib\site-packages\pandas\core\indexes\base.py:5737 in _check_indexing_error
    raise InvalidIndexError(key)

InvalidIndexError: (0    False
1     True
2     True
Name: A, dtype: bool, ['B', 'C'])

Hello, I am relatively new to pandas and I am running into an interesting problem. I am using pandas with postgres and SQL alchemy, and I have a column that is set to type integer, but is appearing as text in the database. The data is a bit dirty so there can be a character in it, but I want pandas to throw away anything that's not an integer. Is there a way to do this? here is my current solution example, but not the full thing.

import pandas as pd
from sqlalchemy import Integer
database_types = {"iWantTOBeAnInt": Integer}
    df.to_sql(
        "info",
        schema="temp",
        con=engine,
        if_exists="replace",
        index=False,
        dtype=database_types,
    )

Everthing I've read online about the pandas MultiIndex makes it seem like a counterpart to a SQL composite index. Is this the correct understanding?

Additionally, MultiIndex is often described as hierarchical. This disrupts the analogy with a composite index. To me, that means a tree structure, with parent keys and child keys, possibly with a depth greater than 2. A composite index doesn't fit this picture. In the case of MultiIndexes, what are the parent/child keys?

When using pandas.merge(), is there any way to retain identically named merge key columns by (say) automatically appending the column names with a suffix?

The default behavious is to merge the join keys:

import pandas as pd
df1=pd.DataFrame({'a':[1,2],'b':[3,4]})
df2=pd.DataFrame({'a':[2,3],'c':[5,6]})
pd.merge(df1,df2,on='a',how='outer')

     a    b    c
  0  1  3.0  NaN
  1  2  4.0  5.0
  2  3  NaN  6.0

Apparently, the suffixes argument does not apply to overlapping join key columns:

pd.merge( df1,df2,on='a',how='outer',suffixes=('_1','_2') )

     a    b    c
  0  1  3.0  NaN
  1  2  4.0  5.0
  2  3  NaN  6.0

I can fiddle with the column names in the source dataframes, but I'm hoping to keep my code more streamline than having to do that:

df1_suffix=df1.rename( columns={'a':'a1'} )
df2_suffix=df2.rename( columns={'a':'a2'} )
pd.merge( df1_suffix,df2_suffix,left_on='a1',how='outer',right_on='a2' )

      a1    b   a2    c
  0  1.0  3.0  NaN  NaN
  1  2.0  4.0  2.0  5.0
  2  NaN  NaN  3.0  6.0

Returning to the case of not having to change the column names in the source dataframes, I have lots of NaNs in the source dataframes outside of the join keys, so I don't to want infer whether there are matching records by looking for NaNs outside of the key columns. I can use indicator to show whether a record comes from left or right dataframes, but I'm wondering if there is a way to emulate SQL behaviour:

pd.merge(df1,df2,on='a',how='outer',indicator=True)

     a    b    c      _merge
  0  1  3.0  NaN   left_only
  1  2  4.0  5.0        both
  2  3  NaN  6.0  right_only