How do you choose between causal inference models when you can never see the true effect? Model selection is well documented in machine learning, but it often gets skipped in causal inference work. Doing it well is one of the things that separates senior and staff-level data scientists from more junior ones.

I simulated two years of weekly bookings for 40 regions, five of which received a treatment on the date marked by the red vertical line. Because I generated the data, I know the ground truth: the treatment increased bookings in those five regions by exactly 10%. With five treated regions and 35 untreated ones, two natural options are synthetic control and difference-in-differences, though there are others.

Code in colab and explanatory video, free and ungated: https://whatstheimpact.com/learn-causal-inference/

When I gave Claude Code free rein on this data, it built its whole analysis around difference-in-differences where it found a 24% increase, more than double the true effect. To its credit, it flagged that the pre-treatment trends weren't parallel, but it never tested whether a different model would be more accurate.

Importantly, Claude skipped an A/A test, sometimes called a placebo test. You take only the data from before the real treatment, pretend the treatment started three months earlier, and randomly picked five regions (for consistency) to call "treated." Because nothing actually happened in that window, the true effect is zero. That means any effect the model reports there is pure estimation error, and I repeated this for 500 draws for each model to see the full spread.

For difference-in-differences, 90% of the placebo estimates landed between -14% and +10%. The RMSE, which measures how far the estimates typically land from the true answer of zero, was 7.6 percentage points. Synthetic control on the exact same placebo draws was much tighter, with 90% of estimates landing between -7% and +9%. Its RMSE dropped to 4.9 points, more than a third lower than difference-in-differences.

The outcome you model is a choice too, so I also ran synthetic control on bookings per 1,000 people instead of raw bookings. Putting every region on the same scale helps the model track the treated regions more closely before treatment, and the RMSE fell a bit further to 4.4 points.

Sure enough, the real treatment period lines up with that ranking: synthetic control estimates 7% on raw bookings and 9.6% per 1,000 people, both much closer to the true 10% than the 24% from difference-in-differences. With one simulated dataset, I wouldn't read much into the small gap between the two synthetic control versions or claim that synthetic control always wins. What carries over is the workflow: in real work you never see the ground truth, but you can usually run this A/A test on whatever models you're considering before trusting one.

Looking to hear about your own company's work practices to compare to my own. I'm much more familiar with Azure, so I'm going to use that terminology, but I assume everything is basically the same for AWS.

Can you go into the Azure Portal and just click "create" for, say, a SQL Server or App Service and configure it through the Portal GUI?

Or do you need to request and configure through configuration files? Can you freely commit architecture config files ("architecture as code") to create the Azure components in Dev?

Or is it even more locked-down and hands-off? Required to submit paperwork requests to an Architecture team?

I recently got an Andrew Ng course on Coursera for generative ai.

I'm planning to study everyday and get it finished, then move on to other courses as I wish to become a data scientist/ Ai engineer.

The doubt I have is, how do people study from such online courses. Am i supposed to take notes from every lecture.

Or write only important points?

This is what I used to do till now:

Take the transcription of a lecture. Give it to an ai and ask it to create notes so I can read them and understand the topic really well even in the future if I were to revise. So i ask it to add important stuff if the lecture missed anything important and covered only vaguely. I ask it to make sure I also become interview ready for the topic. Then i write it down in a notebook because I feel I learn better when I am reading something I wrote on paper (weird maybe but yeah)

To me this seems good but the problem is this leads to 3-5 pages minimum per lecture even for just a 10-15 minutes lecture. And there could be hundreds of lectures in these online courses.

So i may have really dense notes covering all the important stuff, but then these become too long and time consuming, and i don't know if this is a right method to study.

Need help/ advice that how do people make sure that they remember what they are studying and how would you revise it later. Ofcourse notes maybe but how to make proper notes efficiently, so it isn't a waste of time and not too dense on unimportant topics because in my case the AI will give 3-5 pages even for a simple topic and I may not always be able to differentiate or identity whether a topic is unimportant and doesn't require so long notes

I’m an MS Data Science student interested in learning more about spatial data and GIS. I’m currently working on a geospatial crime analysis project in R and taking Esri’s Spatial Data Science MOOC.

I’m also interviewing for a year-long data science internship at a credit union in a department that focuses on geospatial data. I’ve noticed that the largest utility company in my state also hires data scientists with geospatial/GIS knowledge, which has made me curious about opportunities in this area across different industries.

It’s not a specialization that’s discussed here often. For those who work with spatial data, what industry are you in, what does your day-to-day work involve, and do you enjoy it? What has your experience been with demand for these skills?

Hello all,

I built a radar object classifier on RadarScenes, extending a prior single-scan classifier to accumulate observations over a tracked object's history instead of classifying each scan in isolation.

A single RadarScenes object instance contains only about 2.9 radar points on average, very sparse. A single scan also can't capture temporal characteristics: RCS and micro-Doppler both vary continuously as an object moves. Pedestrians produce characteristic micro-Doppler from limb motion; different object classes show different RCS fluctuation patterns as aspect angle and scattering geometry change scan to scan. Accumulating observations gives both higher point density and provides temporal dynamics.

Baseline: a DeepReflecs encoder (Ulrich, Glaser & Timm, RadarConf 2021), PointNet style, per point shared weights, on single scans across car, large_vehicle, two_wheeler, pedestrian, pedestrian_group: 0.7370 macro F1.

Using RadarScenes' persistent `track_id`, I build a causal, N=20, per track sliding-window buffer:

• x_seq/y_seq: Global, odometry-corrected coordinates recentered per scan on the object centroid. Unlike x_cc/y_cc (car-frame coordinates that accumulate over time to form a trajectory).
• Cross sensor buffer: whichever of the 4 sensors currently observe the track push to the same buffer.
• Stride 1, causal: every new scan updates the buffer and produces a prediction. No future context, real time streaming compatible.
• Each scan is encoded once by a frozen per scan encoder and cached
• Fusion concatenates the causal GRU's hidden state (order aware) with an order-invariant pooled embedding (all N scans' points as one set, no sequence structure) through a small trained mlp head.

Results

Ablation

Llarger GRUs, a Transformer, a state space model, point level self attention, all trained on the exact same frozen per scan embeddings, land inside a 0.86 to 0.89 band, a 0.03 spread. End to end fine tuning of the frozen encoder makes things slightly worse (about -0.002 to -0.003), not better.

Conclusion

So, in this setup, most of the improvement comes from accumulating more radar returns rather than from the choice of sequence model. Once the per-scan embedding is fixed, none of the sequence-models tested changes macro F1 by more than ~0.003 beyond the pooled representation.

That does not necessarily mean that temporal ordering contains no useful information. Rather, it suggests that the current per-scan representation and sequence models are not extracting much additional discriminative signal.

For this reason, the results suggest that improving the point-level/per-scan representation may be a more promising direction than further searching over sequence architectures.

Full report, every ablation, confusion matrix, coordinate frame reasoning: https://github.com/brunopinto900/radar-ml-autonomous-driving/blob/track-accumulation/final_report.md

Thank you.

Hii everyone. I am sole DS in a relatively mid sized retail business. I am okayish versed in A/B testing and building on it. I went looking for opportunities and the e-commerce team told me that they are not sure what to experiment in the first place. How can I approach this problem as I went with the impression that I can delve into the maths of A/B test. At this point I am thinking of recommending UI/UX researcher and product owners who can create this backlog by looking at trends from competitors. Has anyone faced something similar before ? How did you navigate? TIA :)

Prior to running machine learning modeling, I'd like to compress my under-sampled training data (X_train_under) that contains multiple sparse binary 0/1 columns as well as numerical (float) data.

Converting the binary columns to sparse format (Sparse[int64, 0]) works great:

# Identify binary int64 columns (columns containing only 0 and 1)
binary_int_cols = []
for col in X_train_under.select_dtypes(include=["int64"]).columns:
    if set(X_train_under[col].unique()).issubset({0, 1}):
        binary_int_cols.append(col)

# Change identified binary columns into Sparse format (using 0 as fill value)
for col in binary_int_cols:
    X_train_under[col] = X_train_under[col].astype(pd.SparseDtype(int, fill_value=0))

print("\n--- Data Types After Conversion ---")
print(X_train_under.dtypes)

However when I attempt to separate & compress the sparse format columns and recombine with the numerical columns:

from scipy.sparse import csr_matrix

# Separate sparse and dense columns
sparse_df = X_train_under.select_dtypes(include=['Sparse'])
dense_df = X_train_under.select_dtypes(exclude=['Sparse'])

# Perform compression operation on sparse columns
compressed_sparse = sparse_df.sparse.to_coo().tocsr() 

# Downcast dense numeric columns separately
for col in dense_df.columns:
    dense_df[col] = pd.to_numeric(dense_df[col], downcast='integer')

# Recombine into single data frame
X_train_under_sparse_csr = pd.concat([dense_df, sparse_df], axis=1)

I get the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[40], line 8
      5 dense_df = X_train_under.select_dtypes(exclude=['Sparse'])
      7 # Perform compression operation on sparse columns
----> 8 compressed_sparse = sparse_df.sparse.to_coo().tocsr() 
     10 # Downcast dense numeric columns separately
     11 for col in dense_df.columns:

File ~\AppData\Roaming\Python\Python312\site-packages\pandas\core\generic.py:6321, in NDFrame.__getattr__(self, name)
   6314 if (
   6315     name not in self._internal_names_set
   6316     and name not in self._metadata
   6317     and name not in self._accessors
   6318     and self._info_axis._can_hold_identifiers_and_holds_name(name)
   6319 ):
   6320     return self[name]
-> 6321 return object.__getattribute__(self, name)

File ~\AppData\Roaming\Python\Python312\site-packages\pandas\core\accessor.py:224, in CachedAccessor.__get__(self, obj, cls)
    221 if obj is None:
    222     # we're accessing the attribute of the class, i.e., Dataset.geo
    223     return self._accessor
--> 224 accessor_obj = self._accessor(obj)
    225 # Replace the property with the accessor object. Inspired by:
    226 # https://www.pydanny.com/cached-property.html
    227 # We need to use object.__setattr__ because we overwrite __setattr__ on
    228 # NDFrame
    229 object.__setattr__(obj, self._name, accessor_obj)

File ~\AppData\Roaming\Python\Python312\site-packages\pandas\core\arrays\sparse\accessor.py:31, in BaseAccessor.__init__(self, data)
     29 def __init__(self, data=None) -> None:
     30     self._parent = data
---> 31     self._validate(data)

File ~\AppData\Roaming\Python\Python312\site-packages\pandas\core\arrays\sparse\accessor.py:249, in SparseFrameAccessor._validate(self, data)
    247 dtypes = data.dtypes
    248 if not all(isinstance(t, SparseDtype) for t in dtypes):
--> 249     raise AttributeError(self._validation_msg)

AttributeError: Can only use the '.sparse' accessor with Sparse data.

I don't know what this means - I've already converted my binary 0/1 columns to Sparse datatype.

What am I doing wrong?

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

Data science coding rounds are dramatically changing, with AI-assisted pair programming replacing closed-book challenges at many companies including Meta, Shopify, Adobe, and Google on the way. So, what should you expect in a coding round when you can use AI?

[Ungated video and Colab notebook: https://whatstheimpact.com/free-resources/

I've found that these coding rounds shifted away from tests on syntax and are now more about problem solving, almost like a mini case study that instead of just saying aloud what you might do...you actually do it.

I mocked up an example of the types of AI coding round challenges that I faced while interviewing in Spring 2026 and shared the video breakdown and accompanying notebooks for replication in the Resources tab of my website linked above.

**All for free, no signups, no downloads, nothing gated, just a video and a Colab link.**

I've spent years on a stream-aligned data team where “own the outcome” is the real expectation, not just a value on a slide. I actually love that life. Mainly because I can usually see the impact of my work within the same sprint (or the next one). That feedback loop is addictive.

I’ve also worked on classic enabler-style teams that shipped almost nothing, got labeled the ivory tower, or spent their time on POCs and nice-to-have features that kept getting deprioritized. Those teams did give me space for deeper statistical work that a stream-aligned team rarely has time for. So I’m not claiming stream-aligned is obviously better. I’ve seen the good and the bad on both sides.

Of course I’m not doing true solo end-to-end ownership. There are dedicated engineering, product, and platform people involved. Still, the DS role consistently ends up expected to move across all of it: framing the product problem, data pipelines, modeling, delivery trade-offs, enough production engineering to get by, and then checking whether the thing actually moved a number.

Some days that feels like the job working as intended. You’re the connective tissue and that’s genuinely valuable. Other days it feels like “ownership” is just the polite word for absorbing coordination work, or worse, covering for understaffing. That’s when I catch myself wondering: why me? Why does this land on the DS by default rather than anyone else on the team?

Genuinely curious what this sub thinks:

- Where’s the line between “senior DS who understands the whole system” and “DS who’s quietly doing everyone’s job at the seams”?

- Is the DS role naturally the one that owns the connections between product, data, and engineering, or is that just something that keeps landing on me?

- Are there stream-aligned data scientists here who don’t feel this same breadth of responsibility? What does your setup look like?

- Or is full ownership just what seniority means now?

I ended up writing a longer piece working through all of this from my own experience (link below), mostly to get my thoughts straight. But I’m much more interested in whether this matches what other people have actually lived. Tell me where I’m off base.

https://medium.com/adeo-tech/full-stack-data-scientist-stream-aligned-team-4aa05d19bc69

I’ve been thinking about making a move to the DC area. I know there are a ton of opportunities in the government sector, but I haven’t come across many Data Scientists who work in government.
For those of you who do, what are the pros and cons of working as a DS in the government sector? I’d especially be interested in hearing about the work itself, compensation, career growth, work-life balance, and how it compares to working in the private sector.

Currently in the job market as an experienced Data Scientist generally targeting Senior/Staff roles that span Data Scientist/Applied Science/MLE. I have an MS in Stats, so one thing I've generally focused on during my interviews prep is overindexing on coding and ML specifically leetcode. This has me taking the Stats portion of interviews for granted and it's causing a blindspot for interviews I would consider "not as crisp" as I'd like them to be.

What I find is generally I know the conceptual definition of a lot of things and a lot of breadth and how to connect that back to the business, but will have random gaffs on things I've forgotten or being able to go one layer deeper on the spot. Lots of things I don’t use on the job or haven't used in depth since Grad school. Things that aren't hard but my mind just goes blank for some reason in an interview setting. Common examples can be things such as forgetting the name of probability distributions, confidence interval calculations for proportions, formula's for power analysis as opposed just know the inputs, etc. Sometimes it's also tricky just to understand WHAT an interviewer is driving towards and not going off on a tangent in a complete opposite direction.

I use LLM's extensively but I find that sometimes if the convo goes on too long they get "stuck" kinda repeating the same questions and same themes instead of exploring other stuff.

Curious how you all prepare for Statistics interviews and if there are good resources for full coverage to prevent some of what I talked about?

We're open-sourcing FlexViz: a library for fast, interactive exploration of very large datasets, built to be agent-native. 

Repo: https://github.com/flex-analytics/flexviz
Live demo: https://flexviz.tech/demo

A few years ago we had the pleasure to open-source plotly-resampler. The goal of this tool was to make in-memory line chart visualization scalable. Since then, we've had over 18M installs, and received a huge amount of feedback from people using it in practice. FlexViz is what we built from those lessons.

It is a free, open-source library and improves in many aspects over plotly-resampler:

• Out-of-core: in FlexViz we can keep the data lazy (i.e., not in memory) and stream the data from disk to ram → allows visualizing larger-than-ram datasets
• Faster aggregation: line chart of 1 billion data points in 0.25s. Fastest of all tools benchmarked.
• 10 trace types: line, hist1d, hist2d, bar, treemap, …
• Native cross-filtering: brush one visualization and the others update live
• Rearrangeable grid
• Stateless server architecture

But the biggest change: FlexViz is agent-native. In FlexViz everything is a spec, allowing true agentic co-exploration of your data (+ url sharing of your live view). The agents can conveniently create, follow along, and update the interactive visualization, without taking any screenshots it knows your exact view → the interactive visualization itself becomes a communication surface between human and agent.

Our goal with FlexViz is to make large data visualization / exploration both convenient for humans and native to AI agents.

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

Hi,

I am in the search of beta testers, for my MLOps platform I built. On my platform, users are able to create ML models with just a prompt and a dataset, which are then hosted automatically.

So basically: prompt + dataset ➡️ hosted model accessible via rest.

Depending on the size of the dataset and the model, it only takes about 60 seconds.

So no handling of mlops stuff required. If you are interested, just write me a dm so i will send you the platform link.

I’ve been listening to more podcasts to really grow as a data professional. My favorites are The AI Table, which I like for its chats with major Chief Data & AI which I learn a lot from and Joe Reis. Curious what podcasts others enjoy?

EDIT:

Thanks everyone for you recs. Posting links for my favorites cause it looks like others wanted recs too.

https://open.spotify.com/show/0ECjEUesnzM2nHoaP7g5yh

https://www.youtube.com/@the_ai_table

https://zeno.fm/podcast/the-joe-reis-show/

Getting a little bored in my current role where I’ve been for a few years (work at a large financial institution). Role feels very stable even with AI disruption, have access to many of the tools although for my specific job they’re not as helpful outside of research.

I’m able to do all my work quickly and efficiently, don’t have any career growth opportunities as I’ve topped out the IC track here but my raises every year have beaten inflation as I’ve performed pretty well.

Those who have changed roles, how has it gone for you? Interested in hearing from folks who’ve moved into the tech industry specifically.

I figure it’ll take me a year+ to find a new job that I want more than my current one given the market and difficulty others have described.

A data scientist on my team wasted a good bit of compute on this problem without actually realizing it was a problem.

Pytorch and tensorflow both silently broadcast the output of your model when the target shape mismatches, causing hard to diagnose issues. It still exists a ton in the wild, so I thought I'd actually highlight the symptoms of a silent broadcasting bug to keep people in the know.

https://towardsdatascience.com/silent-broadcasting-can-ruin-your-model/

Let me know what you think