• bitcoinBitcoin(BTC)$63,681.00-2.07%
  • ethereumEthereum(ETH)$1,788.14-1.82%
  • tetherTether(USDT)$1.00-0.01%
  • binancecoinBNB(BNB)$586.87-0.70%
  • usd-coinUSDC(USDC)$1.000.00%
  • rippleXRP(XRP)$0.96-0.18%
  • solanaSolana(SOL)$72.381.37%
  • tronTRON(TRX)$0.3068720.07%
  • Figure HelocFigure Heloc(FIGR_HELOC)$1.01-3.69%
  • HyperliquidHyperliquid(HYPE)$67.680.57%
  • zcashZcash(ZEC)$915.972.18%
  • moneroMonero(XMR)$505.211.47%
  • dogecoinDogecoin(DOGE)$0.058551-1.52%
  • leo-tokenLEO Token(LEO)$9.301.34%
  • USDSUSDS(USDS)$1.00-0.01%
  • whitebitWhiteBIT Coin(WBT)$61.54-2.00%
  • chainlinkChainlink(LINK)$9.06-1.71%
  • cardanoCardano(ADA)$0.151828-1.30%
  • Ethena USDeEthena USDe(USDE)$1.000.00%
  • stellarStellar(XLM)$0.137011-0.47%
  • daiDai(DAI)$1.00-0.01%
  • bitcoin-cashBitcoin Cash(BCH)$206.74-0.91%
  • nearNEAR Protocol(NEAR)$3.16-0.41%
  • litecoinLitecoin(LTC)$44.25-1.36%
  • RainRain(RAIN)$0.004655-24.60%
  • CantonCanton(CC)$0.080455-2.47%
  • uniswapUniswap(UNI)$5.232.27%
  • avalanche-2Avalanche(AVAX)$7.06-1.48%
  • tether-goldTether Gold(XAUT)$3,839.490.00%
  • the-open-networkGram (prev. Toncoin)(GRAM)$1.04-0.64%
  • suiSui(SUI)$0.71-1.32%
  • USD1USD1(USD1)$1.000.03%
  • Ripple USDRipple USD(RLUSD)$1.000.01%
  • Global DollarGlobal Dollar(USDG)$1.000.01%
  • hedera-hashgraphHedera(HBAR)$0.055992-3.15%
  • Ondo US Dollar YieldOndo US Dollar Yield(USDY)$1.150.03%
  • paypal-usdPayPal USD(PYUSD)$1.000.00%
  • crypto-com-chainCronos(CRO)$0.0433280.51%
  • shiba-inuShiba Inu(SHIB)$0.000004-1.91%
  • BittensorBittensor(TAO)$189.44-0.42%
  • quant-networkQuant(QNT)$151.23-3.74%
  • MemeCoreMemeCore(M)$0.894.52%
  • Circle USYCCircle USYC(USYC)$1.140.01%
  • Spiko Amundi Overnight Swap Fund (EUR)Spiko Amundi Overnight Swap Fund (EUR)(EURSAFO)$1.120.18%
  • okbOKB(OKB)$85.15-0.09%
  • pax-goldPAX Gold(PAXG)$3,843.570.04%
  • aaveAave(AAVE)$110.492.58%
  • Falcon USDFalcon USD(USDF)$1.00-0.01%
  • United StablesUnited Stables(U)$1.000.00%
  • BlackRock USD Institutional Digital Liquidity FundBlackRock USD Institutional Digital Liquidity Fund(BUIDL)$1.000.00%
TradePoint.io
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop
No Result
View All Result
TradePoint.io
No Result
View All Result

Do Modern Problems Require Modern Solutions? This Study Examines the Effectiveness of MLPs (Multi-Layer Perceptions) in Modern Settings

July 11, 2023
in AI & Technology
Reading Time: 5 mins read
A A
Do Modern Problems Require Modern Solutions? This Study Examines the Effectiveness of MLPs (Multi-Layer Perceptions) in Modern Settings
ShareShareShareShareShare

Deep Learning is everywhere. It has revolutionized lots of fields, and the trend seems to continue in the near future. Deep learning is a subfield of machine learning which enables computers to learn and make predictions from large amounts of data. Artificial neural networks are the key component here which are inspired by the structure and functioning of the human brain. 

The ability of deep learning to handle high-dimensional, unstructured data and its remarkable performance in complex tasks have made it a cornerstone of modern AI applications, driving advancements in fields ranging from healthcare and finance to robotics and computer vision.

Deep learning algorithms leverage the power of large-scale computational resources, such as graphics processing units (GPUs), to efficiently train and optimize models with millions or even billions of parameters. Nowadays, these models leverage large, pre-trained models that are fine-tuned for specific tasks. While these models, such as the Transformer architecture for natural language processing and convolutional and transformer-based models for computer vision, have achieved remarkable empirical success, their theoretical understanding lags behind.

[Sponsored] 🔥 Build your personal brand with Taplio  🚀 The 1st all-in-one AI-powered tool to grow on LinkedIn. Create better LinkedIn content 10x faster, schedule, analyze your stats & engage. Try it for free!

The disconnect between theory and practice is growing. This mainly concerns the mathematical simplicity of multi-layer perceptrons (MLPs) that are often used as test beds for analyzing empirical phenomena in more complex models. However, there is a scarcity of empirical data on MLPs trained on benchmark datasets or studies on pre-training/transfer learning. 

This knowledge gap raises a fundamental question: to what extent can MLPs hold on with modern neural network architectures if we were to train them in modern settings? Let’s take a look.

Researchers from ETH Zürich conducted a series of experiments to evaluate MLPs’ performance in modern settings. They also explored the role of inductive bias and the impact of scaling compute resources on the performance of MLPs. By pushing the empirical performance limits of MLPs composed of multiple MLP blocks, the goal here is to bridge the gap between theoretical understanding and practical advancements.

MLPs are known for lacking inductive bias due to their invariance to pixel permutations. Recent advancements, such as the Vision Transformer (ViT) and the MLP-Mixer, challenge the importance of inductive bias in achieving high performance. This study aims to determine whether the absence of inductive bias can be compensated by scaling compute resources, shedding light on the relationship between inductive bias, compute resources, and performance in MLPs.

The study aims to investigate the behavior of MLPs and their empirical performance compared to practical models like Transformers and MLP-Mixers. It also explores the role of inductive bias and the impact of scaling compute resources. The authors hypothesize that MLPs can reflect empirical advances exhibited by practical models and that the lack of inductive bias can be compensated by scaling compute resources.

The research provides several key findings. To start with, it shows that MLPs exhibit similar behavior to their modern counterparts when subjected to scale. Moreover, it highlights the importance of regularization, data augmentation, and the role of SGD’s implicit bias, which differs from CNNs. Surprisingly, larger batch sizes are found to generalize better for MLPs, challenging the role of MLPs as proxies for investigating the implicit bias of SGD. Although the scale employed in this work is insufficient to support the hypothesis of lack of inductive bias being beneficial for performance, it demonstrates that sufficient scale can overcome the bad inductive bias of MLPs, leading to surprisingly strong downstream performance.

In summary, this research demonstrates that even “bad” architectures like MLPs can achieve strong performance when given enough compute resources, with a shift towards investing more compute in dataset size rather than model size.


Check out the Paper and Code. Don’t forget to join our 26k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more. If you have any questions regarding the above article or if we missed anything, feel free to email us at [email protected]

🚀 Check Out 800+ AI Tools in AI Tools Club


YOU MAY ALSO LIKE

iCloud And Other Apple Services Are Down In An Apparent Outage

You.com Launches Extract API for Document Retrieval and PDF Parsing – Unite.AI

Ekrem Çetinkaya received his B.Sc. in 2018, and M.Sc. in 2019 from Ozyegin University, Istanbul, Türkiye. He wrote his M.Sc. thesis about image denoising using deep convolutional networks. He received his Ph.D. degree in 2023 from the University of Klagenfurt, Austria, with his dissertation titled “Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning.” His research interests include deep learning, computer vision, video encoding, and multimedia networking.


🔥 StoryBird.ai just dropped some amazing features. Generate an illustrated story from a prompt. Check it out here. (Sponsored)

Credit: Source link

ShareTweetSendSharePin

Related Posts

iCloud And Other Apple Services Are Down In An Apparent Outage
AI & Technology

iCloud And Other Apple Services Are Down In An Apparent Outage

November 17, 2026
You.com Launches Extract API for Document Retrieval and PDF Parsing – Unite.AI
AI & Technology

You.com Launches Extract API for Document Retrieval and PDF Parsing – Unite.AI

November 17, 2026
NVIDIA Establishes Export Control Committee Following Jensen Huang Resignation
AI & Technology

NVIDIA Establishes Export Control Committee Following Jensen Huang Resignation

November 17, 2026
5 Important Bose Headphone Settings That Need To Be Manually Turned On
AI & Technology

5 Important Bose Headphone Settings That Need To Be Manually Turned On

November 17, 2026
Next Post
Zeta Interactive’s Chief Breaks Down Big Data Competition

Zeta Interactive’s Chief Breaks Down Big Data Competition

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

No Result
View All Result
Why Chase, Microsoft and United are facing crippling outages — and why it’ll get worse

Why Chase, Microsoft and United are facing crippling outages — and why it’ll get worse

November 16, 2026
i-80 Gold: Development Remains On Track & The Stock Offers Great Value (NYSE:IAUX)

i-80 Gold: Development Remains On Track & The Stock Offers Great Value (NYSE:IAUX)

November 16, 2026
Nvidia’s Jensen Huang forfeits 8M – what could happen to the rest of his fortune

Nvidia’s Jensen Huang forfeits $108M – what could happen to the rest of his fortune

November 12, 2026

About

Learn more

Our Services

Legal

Privacy Policy

Terms of Use

Bloggers

Learn more

Article Links

Contact

Advertise

Ask us anything

©2020- TradePoint.io - All rights reserved!

Tradepoint.io, being just a publishing and technology platform, is not a registered broker-dealer or investment adviser. So we do not provide investment advice. Rather, brokerage services are provided to clients of Tradepoint.io by independent SEC-registered broker-dealers and members of FINRA/SIPC. Every form of investing carries some risk and past performance is not a guarantee of future results. “Tradepoint.io“, “Instant Investing” and “My Trading Tools” are registered trademarks of Apperbuild, LLC.

This website is operated by Apperbuild, LLC. We have no link to any brokerage firm and we do not provide investment advice. Every information and resource we provide is solely for the education of our readers. © 2020 Apperbuild, LLC. All rights reserved.

No Result
View All Result
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop

© 2023 - TradePoint.io - All Rights Reserved!