טיוטה:Residual neural network

Residual Neural Network (ידועה גם בשם ResNet; בתרגום חופשי לעברית: רשת עצבית שיורית) היא ארכיטקטורת רשת קונבולוציה שנוצרה על ידיי קימינג הה, צ'יאנגיאו זהנג, שאוצ'ינג רן, וג'יאן סאן.

ResNet התחרתה וזכתה בתחרות "אתגר זיהוי חזותי בקנה מידה גדול" של ImageNet ב-2015. החידוש במשפחת מודלי ה-ResNet הינו החיבורים המדלגים בין כל מודול ומודול ובעצם עוזרים לפתור את בעיית הגרדיאנט הנעלם, נעשה שימוש בחיבורים כאלו גם במודלים נוספים כגון מודלי LSTM ו (BERT, GPT models such as ChatGPT)Transformer models.

כותרת

רקע

מודל הAlexNet היה המודל המנצח ב-"אתגר זיהוי חזותי בקנה מידה גדול" של ImageNet ב-2012 והיה בו כשמונה שכבות קונבולוציה, ב2014 זכה מודל הVGG שהכיל כ-19 שכבות קונבולוציה אך כשניסו החוקרים להוסיך למודל שכבות נוספות הם הבחינו בירידה במידת הדיוק של המודל.

Background[edit source]

The AlexNet model developed in 2012 for ImageNet was an 8-layer convolutional neural network. The neural networks developed in 2014 by the Visual Geometry Group (VGG) at the University of Oxford approached a depth of 19 layers by stacking 3-by-3 convolutional layers. But stacking more layers led to a quick reduction in training accuracy, which is referred to as the "degradation" problem.

A deeper network should not produce a higher training loss than its shallower counterpart, if this deeper network can be constructed by its shallower counterpart stacked with extra layers. If the extra layers can be set as identity mappings, the deeper network would represent the same function as the shallower counterpart. It is hypothesized that the optimizer is not able to approach identity mappings for the parameterized layers.

Residual Learning[edit source]

In a multi-layer neural network model, consider a subnetwork with a certain number (e.g., 2 or 3) of stacked layers. Denote the underlying function performed by this subnetwork as , where  is the input to this subnetwork. The idea of "Residual Learning" re-parameterizes this subnetwork and lets the parameter layers represent a residual function . The output  of this subnetwork is represented as:

This is also the principle of the 1997 LSTM cell computing , which becomes  during backpropagation through time.

The function  is often represented by matrix multiplication interlaced with activation functions and normalization operations (e.g., Batch Normalization or Layer Normalization).

This subnetwork is referred to as a "Residual Block". A deep residual network is constructed by stacking a series of residual blocks.

The operation of "" in "" is approached by a skip connection that performs identity mapping and connects the input of a residual block with its output. This connection is often referred to as a "Residual Connection" in later work.

Signal Propagation[edit source]

The introduction of identity mappings facilitates signal propagation in both forward and backward paths.

Forward Propagation[edit source]

If the output of the -th residual block is the input to the -th residual block (i.e., assuming no activation function between blocks), we have:

Applying this formulation recursively, e.g., , we have:

where  is the index of any later residual block (e.g., the last block) and  is the index of any earlier block. This formulation suggests that there is always a signal that is directly sent from a shallower block  to a deeper block .

Backward Propagation[edit source]

The Residual Learning formulation provides the added benefit of addressing the vanishing gradient problem to some extent. However, it is crucial to acknowledge that the vanishing gradient issue is not the root cause of the degradation problem, as it has already been tackled through the use of normalization layers. Taking the derivative w.r.t.  according to the above forward propagation, we have:

Here  is the loss function to be minimized. This formulation suggests that the gradient computation of a shallower layer  always has a term  that is directly added. Even if the gradients of the  terms are small, the total gradient  is not vanishing thanks to the added term

ראו גם

• שם ערך

לקריאה נוספת

• שם סופר, שם ספר, שם הוצאה, תאריך הוצאה

קישורים חיצוניים

• התוכן בקישור, באתר (שם האתר)