• Contents.

Motivation

2025.01.04.

I am motivated again recently by the SciForDL workshop at NeurIPS 2024. There is an interesting part of the workshop named ‘Debunking Challenge’, where the webpage says that:

“We are hosting a challenge to interrogate commonly-held beliefs in the deep learning community. Relevant submissions will challenge machine learning theory, common assumptions and/or folk knowledge.”

<aside> 📔

Debunk: verb, expose the falseness or hollowness of (a myth, idea or belief).

</aside>

The requirements of this challenge are:

• What commonly-held position or belief are you challenging?
• How are your results in tension with this commonly-held position?
• How do you expect your submission to affect future work?

The webpage also gives three types of good debunking:

• To challenge “Experiments or conclusions drawn from experiments”, e.g. the Chinchilla scaling law figure.
• To challenge “Beliefs motivated by theoretical results”, e.g. theories suggesting deep learning models generalize because they cannot fit random labels.
• To challenge “Common assumptions or folk-wisdom”, e.g. conjectures that suggests the 1d subspace of feature representation in language models.

Currently (25.01.04), my proudest belief of deep learning is the following argument motivated through the double descent paper:

<aside> 💡

Given a task, we can tackle it with neural network models to get better performance (generalization ability) by simply increasing the capacity of the model. Increasing the model capacity is doomed to work and the key to success is to do extensive hyperparameter tuning to make optimization work in the first place.

</aside>

However, recently I become skeptical towards the above tenet due to the following experience.

• [E1] Increasing Depth Leads to U-Shaped Test Risk in Over-parameterized Convolutional Networks, Jun. 4 2021.
• [E2] My own experience on increasing capacity of a MLP for classifying industrial products.

In E1, I find that paper which observes that when increasing the depth of a convolutional ResNet, the test error curve is still U-shape as in traditional machine learning theory, instead of a double-descent curve, at least on CIFAR10 image classification dataset.