tzyt's Blog

Understanding the Least-Squares Method From the Perspectives of Calculus and Linear Algebra

2023-12-08T20:48:06.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

I previously used the least-squares method when trying to implement some computer graphics papers, but at that time I understood it from the perspective of calculus. This year, after taking a linear algebra course, I found that it can also be explained using concepts from linear algebra and that this explanation is more intuitive (it provides a geometric understanding). Therefore, I plan to use this article to record two ways of understanding the least-squares method.

Before beginning, I first want to complain about the Chinese name of this method, “minimum two multiplication.” It just feels very strange: why is minimizing the square of the error called “two multiplication”? Well… after looking it up, I found that it was translated from Japanese, so I can only say that the quality of this translation is rather impressive… Although I really do not want to use this confusing name, there is nothing I can do because everyone uses it.

Definition of the Problem

Before learning the least-squares method, we first need to understand what problem this algorithm is trying to solve. The most common use of the least-squares method is fitting data with a function and then using that function to predict the trend of the data. To fit data, we need to use a mathematical method to define what constitutes a good fit, and then try to make a function fit the data better.

Here, suppose there are $m$ data points $(x_i, y_i) \ \ i \in \{0, 1, \cdots, m\}$ . There is also a function $f(x)$ used to fit these data points. We define the error of an individual point as:

$s_i = f(x_i) - y_i$

Note that the meaning of the function $f(x_i)$ here is using $x_i$ to predict $y_i$ . You can understand these two values as scalars, but the least-squares method can actually also be used to fit vectors, because the formulas are exactly the same in both cases. For simplicity, we assume here that they are scalars.

Then the sum of squared errors is:

$I = \sum_{i=0}^m s_i^2 = \sum_{i=0}^m (f(x_i) - y_i)^2$

We want to adjust the parameters of this function to minimize this value.

Specifically, the function $f(x)$ can be expressed in the following form:

$f(x) = a_0\varphi_0(x) + a_1\varphi_1(x) + \cdots + a_n\varphi_n(x)$

Here, $a_i$ are the parameters that we need to adjust, while $\varphi_i(x)$ are some linearly independent functions.

Calculus Perspective

The content of this section mainly comes from this video. Its content is quite complete, but some parts move quickly and contain writing mistakes. I also watched it several times before understanding it.

We can express the error in the following form and then minimize this error from the perspective of calculus.

$\begin{align*}I(a_0, \cdots, a_n) &= \sum_{i=0}^m w(x_i) (f(x_i) - y_i)^2 \\I(a_0, \cdots, a_n) &= \sum_{i=0}^m w(x_i)\left(\sum_{j=0}^n [a_j\varphi_j(x_i)] - y_i\right)^2 \end{align*}$

Writing it in this form expresses our objective more clearly: minimizing the error by adjusting the parameters $a_0, \cdots, a_n$ .

Note that, when summing the squared error of each data point, an additional weight $w(x_i)$ appears here. This weight makes it more convenient to adjust the importance of each data point.

From calculus, we know that the derivative of a function must be 0 when the function reaches an extremum. We can use this property to adjust the parameters from the perspective of calculus so that the error is minimized. However, this extremum may be either a minimum or a maximum. When calculating the error $I$ , only one minimum can exist (you can imagine the situation in which all parameters are set to $\infty$ ).

Because $I$ is a multivariable function, we need to use partial derivatives to find the minimum with the method above:

$\frac{\partial I}{\partial a_k} = 2\sum_{i=0}^m w(x_i)\left(\sum_{j=0}^n[ a_j\varphi_j(x_i)] - f(x_i)\right)\varphi_k(x_i) = 0 \\$

In the partial derivative above, everything other than $a_k$ can be treated as a constant. This is because the definition of a partial derivative is the effect on the error $I$ caused by changing the parameter $a_k$ .

When $\frac{\partial I}{\partial a_k} = 0$ , we can say that, when only the parameter $a_k$ is adjusted, the error has already reached its minimum. However, because every parameter can be adjusted, we want $\frac{\partial I}{\partial a_k} = 0$ for every $k \in \{0, \cdots, n\}$ .

In this way, we obtain a system of linear equations. When seeing a system of linear equations, the first thought is certainly to represent it using linear algebra, which can greatly increase the solution speed.

After some transformations, we obtain the following formula:

$\begin{align*}0 &= 2\sum_{i=0}^m w(x_i)\left(\sum_{j=0}^n[ a_j\varphi_j(x_i)] - f(x_i)\right)\varphi_k(x_i)\\ &= 2\sum_{i=0}^m\left(w(x_i)\varphi_k(x_i)\sum_{j=0}^n\left[a_j\varphi_j(x_i)\right] - w(x_i)y_i\varphi_k(x_i) \right) \\ &= \sum_{i=0}^m\left(w(x_i)\varphi_k(x_i)\sum_{j=0}^n\left[a_j\varphi_j(x_i)\right]\right) - \sum_{i=0}^m\left[w(x_i)y_i\varphi_k(x_i)\right]\end{align*}$

Because we want to move toward a linear-algebra representation, we can express the following summation as a dot product:

$\sum_{i=0}^m\left[w(x_i)y_i\varphi_k(x_i)\right] \\= \vec{w} \cdot \vec{y} \cdot \vec{\varphi_k}$

Here, $\vec{w} = w(x_i) \ i \in \{0, \cdots, m\}$ , $\vec{y} = y_i \ i \in \{0, \cdots, m\}$ , and $\vec{\varphi_k} = \varphi_k(x_i) \ i \in \{0, \cdots, m\}$ .

Similarly, the other part of the expression can also be represented as a dot product:

$\sum_{i=0}^m\left(w(x_i)\varphi_k(x_i)\sum_{j=0}^n\left[a_j\varphi_j(x_i)\right]\right) \\= \vec{w} \cdot \vec{\varphi_k} \cdot \sum_{j=0}^n \vec{a_j} \cdot \vec{\varphi_j} \\= \sum_{j=0}^n \vec{w} \cdot \vec{\varphi_k} \cdot \vec{\varphi_j} \cdot \vec{a_j}$

Then, if:

$\vec w \cdot \vec y \cdot \vec \varphi_k = \sum_{j=0}^n \vec{w} \cdot \vec{\varphi_k} \cdot \vec{\varphi_j} \cdot \vec{a_j}$

the partial derivative of the error is 0; that is, $\frac{\partial I}{\partial a_k} = 0$ .

Of course, our final goal is to write this expression as a matrix-multiplication equation to increase the solution speed. Looking carefully at the right-hand side of the equation, we can actually see that this summation is essentially also a dot product.

$\sum_{j=0}^n {\color{red}\vec{w} \cdot \vec{\varphi_k} \cdot \vec{\varphi_j}} \cdot \vec{a_j}$

To make this easier to understand, denote the part marked in red as $\vec A_k = [\vec{w} \cdot \vec{\varphi_k} \cdot \vec{\varphi_0}, \cdots, \vec{w} \cdot \vec{\varphi_k} \cdot \vec{\varphi_n}]$ . It is an $n$ -dimensional vector, so this summation is $\vec A_k \cdot \vec a$ , where $a = [a_0, \cdots, a_n]$ .

Similarly, we can denote $\vec w \cdot \vec y \cdot \vec \varphi_k$ as $B_k$ (a scalar).

Thus, solving the following system of equations solves the least-squares problem:

$\begin{cases} \vec A_0 \cdot \vec a = B_0 \\ \vec A_1 \cdot \vec a = B_1 \\ \qquad \vdots \\ \vec A_m \cdot \vec a = B_m\end{cases}$

This form is very familiar and can be represented directly in matrix form.

$\begin{bmatrix} A_{00} & A_{01} & \cdots & A_{0n} \\ A_{10} & A_{11} & \cdots & A_{1n} \\ \vdots & & \ddots & \\ A_{m0} & A_{m1} & \cdots & A_{mn} \\\end{bmatrix} \cdot \begin{bmatrix} a_0 \\ a_1 \\ \vdots \\ a_n \\\end{bmatrix} = \begin{bmatrix} B_0 \\ B_1 \\ \vdots \\ B_m \\\end{bmatrix}$

Written more simply, this is $\bm A \vec a = \vec B$ .

Linear Algebra Perspective

To be completed.

GAMES101 Study Notes 1

2023-06-16T19:30:42.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Because of various matters at school, along with reviewing for and taking exams, it has already been more than four months since I last updated the blog, and even longer since I last wrote a computer graphics post.

Summer vacation recently began, and I finally studied the entirety of GAMES101 after previously watching only the ray-tracing section. It was still a very pleasant surprise: after some time had passed, I gained new understandings of many concepts that I had not understood very clearly before, especially the mathematical ones. Because of the course’s time constraints, some material was not explained in much detail, so I am recording some of my own understanding here.

Three-Dimensional Rotation Matrices

I already wrote about this in the earlier RT: The Next Week article, but the previous explanation was rather… strange and verbose, and it did not explain the subject from the perspective of coordinate-system transformations. I will therefore write it again here (of course, I still have not studied linear algebra systematically, so the following content may still be rather dubious).

The three-dimensional rotation matrices around the three axes can respectively be written in the following forms:

$R_x(\theta) = \begin{bmatrix}1 & 0 & 0 \\0 & \cos \theta & -\sin \theta \\0 & \sin \theta & \cos \theta\end{bmatrix}$

$R_z(\theta) = \begin{bmatrix}\cos \theta & -\sin \theta & 0 \\\sin \theta & \cos \theta & 0 \\0 & 0 & 1\end{bmatrix}$

$R_y(\theta) = \begin{bmatrix}\cos \theta & 0 & \sin \theta \\0 & 1 & 0 \\-\sin \theta & 0 & \cos \theta\end{bmatrix} \\$

It is not difficult to notice that, in the matrix for rotation around the y-axis, the positions of $\sin \theta$ and $-\sin \theta$ seem to be reversed, which is very strange. In the original video, the rotation matrix is derived by selecting several special points. I feel that a coordinate-system transformation makes it easier to understand here (~~although Professor Yan thinks this approach is more complicated~~).

First consider rotation around the z-axis. This case is relatively simple and is basically the same as a two-dimensional rotation matrix:

Note that although I did not draw the z-axis here, the right-hand rule tells us that the z-axis points outward through the screen.

We can express the new x-axis ( $\hat i$ ) and the new y-axis ( $\hat j$ ) separately as vectors. Note that both vectors are unit vectors:

$\hat i = \begin{bmatrix}\cos \theta \\\sin \theta \\0\end{bmatrix} \\$

Observe that the angle between $\hat j$ and the original y-axis is $\theta$ , so we can “reverse” the vector form of $\hat i$ . One more point requires attention: the x-component of $\hat j$ is negative:

$\hat j = \begin{bmatrix}-\sin \theta \\\cos \theta \\0\end{bmatrix} \\$

In fact, in any coordinate system, every coordinate is obtained by multiplying unit vectors by certain lengths. You can imagine this as moving a point a certain distance in a certain direction. For example, in a Cartesian plane, the coordinate $(1, 2)$ can be understood as moving a point 1 unit in the x direction and 2 units in the y direction.

Therefore, in the rotated coordinate system, the new coordinate of a point $(x, y, z)$ is $(x\hat i, y\hat j, z)$ . This is equivalent to moving x units in the $\hat i$ direction, y units in the $\hat j$ direction, and z units in the original z direction. Although that direction was not transformed, we still denote it by $\hat k$ here.

The new coordinate is therefore:

$p = x\begin{bmatrix}\cos \theta \\\sin \theta \\0\end{bmatrix} + y\begin{bmatrix} -\sin \theta \\\cos \theta \\0\end{bmatrix} + z\begin{bmatrix}0 \\0 \\1\end{bmatrix}$

Looking at the rotation-matrix formula given earlier, we can see that $R_z$ indeed satisfies the expression above:

$\begin{bmatrix}\textcolor{red}{\cos \theta} & \textcolor{blue}{-\sin \theta} & \textcolor{green}{0} \\\textcolor{red}{\sin \theta} & \textcolor{blue}{\cos \theta} & \textcolor{green}{0} \\\textcolor{red}{0} & \textcolor{blue}{0} & \textcolor{green}{1}\end{bmatrix}\begin{bmatrix} x \\ y \\ z \\\end{bmatrix} = x\textcolor{red}{\begin{bmatrix}\cos \theta \\\sin \theta \\0\end{bmatrix}} + y\textcolor{blue}{\begin{bmatrix} -\sin \theta \\\cos \theta \\0\end{bmatrix} } + z\textcolor{green}{\begin{bmatrix}0 \\0 \\1\end{bmatrix}}$

There is a rather interesting point here, marked with the different colors: the three columns of the rotation matrix correspond respectively to $\hat i$ , $\hat j$ , and $\hat k$ , which are the directions of the three axes in the new coordinate system.

From the perspective of coordinate-system transformations, this makes it very easy to understand why the rotation matrix is written this way. We can directly put the directions of the three transformed axes into the three columns of the matrix, thereby obtaining the rotation matrix.

We can use the same method to analyze rotation around the y-axis—the matrix that appeared to be “reversed.”

Although this looks very similar to the preceding figure, notice that the labels in the diagram have changed. Again, the right-hand rule tells us that the y-axis now points outward through the screen.

With the preceding observation, we now need only find $\hat i$ , $\hat j$ , and $\hat k$ in the current situation to write down the rotation matrix around the y-axis.

First, for the new z-axis, namely $\hat k$ , we can draw an analogy with $\hat i$ in the z-axis rotation. Its vector form is:

$\hat k = \begin{bmatrix}\sin \theta \\0 \\\cos \theta\end{bmatrix} \\$

The reason the y-component is zero is obvious: we are considering rotation around the y-axis, so y certainly does not change.

Next, by analogy with $\hat j$ in the z-axis rotation, we obtain the vector form of $\hat i$ :

$\hat i = \begin{bmatrix}\cos \theta \\0 \\-\sin \theta\end{bmatrix} \\$

Because $\hat j$ is unchanged from before, it can simply be written as:

$\hat j = \begin{bmatrix}0 \\1 \\0\end{bmatrix} \\$

Combining these vectors that represent the new x, y, and z directions gives:

$\left[\hat i \ | \ \hat j \ | \ \hat k\right] = \begin{bmatrix}\cos \theta & 0 & \sin \theta \\0 & 1 & 0 \\-\sin \theta & 0 & \cos \theta\end{bmatrix} \\$

Barycentric Coordinates

As explained in the GAMES101 course, barycentric coordinates—especially those for triangles—are extremely useful in computer graphics. They make it convenient to interpolate information at a triangle’s vertices across the triangle’s surface.

When I first listened to the lecture, I still had many questions. For example, why must the three coefficients sum to 1 and all be nonnegative for a point to lie inside the triangle?

After thinking about it for some time, I felt that understanding the concept according to the literal meaning of its name was intuitive: understand it from the perspective of a center of mass.

Physical Perspective

Suppose there is a triangle whose three vertices have masses $M_a, M_b, M_c$ , respectively, while every region other than the vertices has mass 0. According to the definition of a center of mass, the triangle’s center of mass is:

$\text{center of mass} = \frac{M_aA + M_bB + M_cC}{M_a + M_b + M_c}$

Separating these three terms gives:

$A\frac{M_a}{M_a + M_b + M_c} + B\frac{M_b}{M_a + M_b + M_c} + C\frac{M_c}{M_a + M_b + M_c}$

This form is extraordinarily similar to the barycentric-coordinate expression $\alpha A + \beta B + \gamma C$ : every term is a coefficient multiplied by a vertex coordinate.

Looking at a term of the form $\frac{M_a}{M_a + M_b + M_c}$ , we find that the coefficients used to calculate the center of mass satisfy the barycentric-coordinate requirements exactly: $\alpha + \beta + \gamma = 1$ , and no term is negative.

Thinking in terms of the physical center of mass, what exactly are we converting when we transform ordinary coordinates into barycentric coordinates?

Map the coefficients in the center-of-mass formula to those in barycentric coordinates, namely $\frac{M_a}{M_a + M_b + M_c} \to \alpha$ . We can then see that these coefficients are actually the proportions of the triangle’s total mass assigned to each vertex.

In other words, during the conversion—suppose we convert point $p$ from Cartesian coordinates to barycentric coordinates—we are really solving this problem: how should weight be distributed among the triangle’s three vertices so that the triangle’s center of mass lies at point $p$ ?

The converted coordinate $(\alpha, \beta, \gamma)$ is precisely the proportion of weight assigned to each of the triangle’s three vertices.

This makes it easy to understand why the three numbers must sum to 1 and be nonnegative for a point to lie inside the triangle. First, from a physical perspective, if mass is nonnegative, the center of mass must lie inside the object. Second, to satisfy the definition of a center of mass, $\alpha$ , $\beta$ , and $\gamma$ must sum to 1. Since they represent the proportions of the total mass at the three vertices, the three numbers necessarily sum to 1, corresponding to the total mass.

Of course, barycentric coordinates are a generalization of the physical definition, so a purely algebraic explanation may be more convincing.

Algebraic Perspective

We know that a triangle’s barycentric coordinates use a linear combination of its three vertices to represent a coordinate: $p = \alpha A + \beta B + \gamma C$ . We can now try to explain algebraically why the three coefficients must sum to 1 and be nonnegative for the point to lie inside the triangle.

First, we can express point $p$ in another form that will help with the later derivation:

$p = A + u\overrightarrow{AB} + v\overrightarrow{AC}$

Note that $u$ and $v$ have no relation to the three barycentric-coordinate coefficients here. This expression means that, starting from point A, we move $u$ units along the direction AB and then $v$ units along the direction AC to reach point $p$ .

First, one thing is certain if the point is to remain inside the triangle: $u$ and $v$ are nonnegative. Starting at A, moving any distance only in the direction $\overrightarrow{BA}$ or $\overrightarrow{CA}$ would immediately leave the triangle.

Rearranging the preceding expression slightly gives:

$p - A = u\overrightarrow{AB} + v\overrightarrow{AC} \\\overrightarrow{pA} = u\overrightarrow{AB} + v\overrightarrow{AC} \\u\overrightarrow{AB} + v\overrightarrow{AC} - \overrightarrow{pA} = {0}$

This expression can be understood as follows: we move every point, including point $p$ , by $-\overrightarrow{OA}$ . At this point, A must lie at the origin.

The expression then becomes intuitive. First, the vector $u\overrightarrow{AB} + v\overrightarrow{AC}$ takes us from the origin to point $p$ ; then $\overrightarrow{pA}$ takes us from point $p$ back to A, which is the origin.

Because these two parts point in opposite directions and have the same magnitude—one travels from A to $p$ , and the other from $p$ to A—the sum of the two vectors must be 0.

Expanding and rearranging the expression gives:

$\begin{align*}u(B - A) + v(C - A) + A - p &= 0 \\uB - uA + vC - vA + A - p &= 0\\A(1 - u - v) + uB + vC - p &= 0 \\(1 - u - v)A + uB + vC &= p\end{align*}$

This form is exactly the same as the barycentric-coordinate expression, so we can determine that $\alpha = 1 - u - v$ , $\beta = u$ , and $\gamma = v$ .

Adding these quantities gives $1 - u - v + u + v = 1$ . Indeed, the three barycentric-coordinate coefficients necessarily sum to 1.

We already established that $u$ and $v$ are nonnegative, but if $u + v > 1$ , would that not make $\alpha$ negative?

Consider the following figure. $AM$ is a perpendicular from $A$ to $BC$ ( $M$ is not the midpoint of $BC$ ; using a non-equilateral triangle might have made this clearer, ~~but I was lazy~~):

Return to the original definition, $p = A + u\overrightarrow{AB} + v\overrightarrow{AC}$ . Here, $u$ and $v$ are respectively the distances traveled in the $\overrightarrow{AB}$ and $\overrightarrow{AC}$ directions.

The length of the projection of $u\overrightarrow{AB}$ onto $\overrightarrow{AM}$ is therefore $u\overrightarrow{AB} \cdot \overrightarrow{AM} \over |\overrightarrow{AM}|$ .

The figure shows that if $u$ is 1—that is, if $\overrightarrow{AB}$ has its full length—the projection’s length must equal $|\overrightarrow{AM}|$ . Likewise, if $u$ is 0, the projected length in the $\overrightarrow{AM}$ direction must also be 0.

Because projection, or equivalently the dot product, is a linear operation, if $u$ is 0.5, the projection’s length must be half of $|\overrightarrow{AM}|$ .

We can therefore say that the length of the projection of $\overrightarrow{AB}$ onto $\overrightarrow{AM}$ is $u|\overrightarrow{AM}|$ .

Similarly, the length of the projection of $\overrightarrow{AC}$ in the $\overrightarrow{AM}$ direction follows the same rule.

For point $p$ to remain inside the triangle, the length of the projection of the vector $u\overrightarrow{AB} + v\overrightarrow{AC}$ onto $\overrightarrow{AM}$ must be no greater than the length of $\overrightarrow{AM}$ itself. Otherwise, point $p$ would cross side $BC$ and leave the triangle.

The projected length of $\overrightarrow{AB}$ onto $\overrightarrow{AM}$ is $u|\overrightarrow{AM}|$ , while the projected length of $\overrightarrow{AC}$ onto $\overrightarrow{AM}$ is $v|\overrightarrow{AM}|$ . Naturally, the projected length of the sum of the two vectors onto $\overrightarrow{AM}$ is $(u + v)|\overrightarrow{AM}|$ .

As stated above, for point $p$ to stay inside the triangle, the projection of $u\overrightarrow{AB} + v\overrightarrow{AC}$ onto $\overrightarrow{AM}$ must be no longer than $\overrightarrow{AM}$ itself. That is, $(u + v)|\overrightarrow{AM}| \le |\overrightarrow{AM}|$ , so $u + v \le 1$ .

Because $\alpha = 1 - u - v$ and $u + v \le 1$ , we can show that $\alpha$ must also be nonnegative.

At this point, we have explained why a point inside a triangle must satisfy $\alpha + \beta +\gamma = 1$ , with every coefficient nonnegative. Of course, converting a Cartesian coordinate into barycentric coordinates still requires some relatively complicated calculations. For this part, I think the first method described in this blog post is relatively easy to understand and quite ingenious; interested readers can take a look.

USACO23JAN Find and Replace S (Luogu P9013) Solution

2023-02-05T21:15:37.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Problem link

The reading experience is better on the blog

Analysis

The statement is very concise: given a series of character replacements, find the minimum number of steps required to turn string $s$ into string $t$ .

After receiving the problem, we can first analyze the samples.

From the sample $\texttt{BBC} \to \texttt{ABC}$ , we can see that it is impossible to replace one character with two characters at the same time ( $\texttt{BB} \to \texttt{AB}$ ), because this creates a conflict.

Then, can the number of positions where $s_i \ne t_i$ (after deduplicating the strings, so cases such as $s = \texttt{AA}, t = \texttt{BB}$ do not exist) directly be used as the answer? The final sample shows that this is not the case.

Handling Cycles

Because the $\texttt{CD}$ part in the final sample is the same, let us directly consider the transformation $\texttt{AB} \to \texttt{BA}$ . If we directly perform the operation $\texttt{A} \to \texttt{B}$ , we obtain a string $\texttt{BB}$ . At this point, the same problem as before appears: it cannot be transformed into $\texttt{BA}$ . Performing $\texttt{B} \to \texttt{A}$ is analogous.

The solution is to first perform $\texttt{AB} \to x\texttt{B}$ and then process $x\texttt{B} \to \texttt{BA}$ ( $x$ is any other character).

Can all mutually dependent cases be solved in this way? Let us consider a larger sample, $\texttt{ABCD} \to \texttt{BCDA}$ . It is clearer to represent it as a graph (create an edge $s_i \to t_i$ and remove duplicate edges and self-loops):

    graph LR    A --> B    B --> C    C --> D    D --> A

This is a cycle. No matter which $x \to y$ transformation we perform first, we will need to perform a $y \to z$ transformation afterward, because $y$ wants to become something else. At this point, the earlier $x$ will also be changed to $z$ .

However, the problem can be solved if we can “turn the cycle into a chain.” For example, we can first perform $\texttt{A} \to x$ , turning the chain into:

    graph LR    x --> B    B --> C    C --> D    D --> A

Now there is a place where, after executing $x \to y$ , we do not need to execute $y \to z$ : $\texttt{D} \to \texttt{A}$ . (After executing it, $\texttt{C} \to \texttt{D}$ also satisfies this condition, so we can convert the entire string to the target by following the chain backward.)

These two examples show that, in general, one operation can turn a cycle into a chain, or reduce the length of a chain (the number of edges) by 1.

So is the answer the number of cycles plus the length of the chains?

Two Special Cases

1

First, turning a cycle into a chain requires a character that does not appear in the cycle. If the cycle contains every character in the character set, we cannot handle it.

Suppose our character set contains only the four characters $\texttt{A} \sim \texttt{D}$ . When processing the following example, we encounter a problem:

    graph LR    A --> B    B --> C    C --> D    D --> A

No matter which character we change $\texttt{A}$ into first, that character will undergo at least one more transformation afterward, preventing $\texttt{A}$ from being transformed into the target character $\texttt{B}$ .

Of course, an unprocessable case does not necessarily require the entire graph to contain only one cycle. It is enough that:

Every node is in a cycle.
Every character in the character set is used.

For example, the following case with two cycles is also impossible (the character set is $\texttt{A} \sim \texttt{C}$ ):

    graph LR    A --> B    B --> A    C --> D    D --> C

2

Consider the input $\texttt{ABCDEF} \to \texttt{BCDABE}$ :

    graph LR    A --> B    B --> C    C --> D    D --> A    E --> B    F --> E

We can turn the cycle into a chain and reduce the chain length by 1 in a single operation. Observe that both $\texttt{A}$ and $\texttt{E}$ want to be transformed into $\texttt{B}$ . From the perspective of character transformations, $\texttt{A} \to \texttt{B} \And \texttt{E} \to \texttt{B}$ and $\texttt{A} \to \texttt{E} \And \texttt{E} \to \texttt{B}$ have the same final result and number of steps. However, when executing $\texttt{A} \to \texttt{E}$ , the second method also transforms a character in the cycle into a character outside the cycle, turning the cycle into a chain.

The prerequisite for doing this is that multiple characters outside the cycle want to become one character inside the cycle. More precisely, some node in the cycle must have in-degree at least 2.

At this point, all cases have essentially been analyzed, and we can summarize them as follows (the conditions in parentheses are the actual checks):

If one character wants to be transformed into multiple characters, there is no solution. (Every node has out-degree at most 1.)
If all nodes (all possible characters) are in cycles, there is no solution. (Every character has in-degree 1.)
Answer = number of edges + number of absolute cycles (every node in the cycle has both in-degree and out-degree equal to 1).

The check for the second point can be explained slightly. We do not choose to use out-degree because of the case where a cycle is connected to a tree; see the figure above.

Code Implementation

When implementing this, pay attention to the cycle-finding part; the other parts are relatively simple.

We know that Tarjan’s algorithm can detect cycles. However, this problem can use a “simplified version” of Tarjan that does not record discovery timestamps. During DFS, we push every visited node from the back of a deque.

If DFS starts from a node on a cycle, it must eventually visit a node equal to the front of the deque. At this point, popping all nodes between the front and back gives all nodes in the cycle.

If we find that a node was visited previously but is not at the front, we can determine that the nodes in the deque are not an “absolute cycle,” because a tree is connected to it (as in the figure above, this happens if the search starts from node F).

#include 
using namespace std;

const int CHSZ = 52;  // Character-set size.
int out[CHSZ + 1];   // The out-degree can only be one.
int lpid[CHSZ + 1];  // Cycle ID: unknown -> -1, not a cycle -> 0, cycle -> 1, 2, 3, ...
enum LP_STAT { UNKNOWN = -1, NOT_ABS_LP = 0 };
deque<int> vised_dq;   // Store information while finding cycles.
bool vised[CHSZ + 1];  // Store information while finding cycles.

set<int> in_nds[CHSZ + 1]; // Incoming nodes; the in-degree can be multiple.
int in1_cnt = 0; // Number of nodes with in-degree 1.

int abs_lp_cnt = 0;  // Number of absolute cycles, namely cycles with no tree attached.
int diff_chs = 0;

void init() {
    memset(out, 0, sizeof(out));
    fill(lpid, lpid + CHSZ + 1, UNKNOWN);
    vised_dq.clear();
    memset(vised, 0, sizeof(vised));
    for (int i = 0; i <= CHSZ; i++) in_nds[i].clear();
    in1_cnt = 0;
    abs_lp_cnt = 0;
    diff_chs = 0;
}

inline int ch2id(char x) {
    // Convert a character to an ID.
    if (x >= 'a' && x <= 'z') return x - 'a' + 1;
    if (x >= 'A' && x <= 'Z') return x - 'A' + 27;
    return -1;
}

bool check_loop_connect_to_tree() {
    for (int cur : vised_dq) 
        if (in_nds[cur].size() >= 2) 
            // A tree is connected to this cycle.
            return true;
    return false;
}

void fill_lpid_in_vised_dq(int val) {
    for (int cur : vised_dq)
        lpid[cur] = val;
    vised_dq.clear();
}

void mark_loop(int cur) {
    if (vised[cur] && vised_dq.front() != cur) {
        // A cycle entered from a tree is not an absolute cycle.
        fill_lpid_in_vised_dq(NOT_ABS_LP);
        return;
    }
    vised[cur] = true;
    if (out[cur] == cur) {
        // No outgoing edge; a chain has been found.
        fill_lpid_in_vised_dq(NOT_ABS_LP);
        return;
    }

    if (vised_dq.size() && vised_dq.front() == cur) {
        // A cycle has been found.
        if (!check_loop_connect_to_tree()) {
            // If no tree is connected to the cycle.
            abs_lp_cnt++;
            fill_lpid_in_vised_dq(abs_lp_cnt);
        } else {
            fill_lpid_in_vised_dq(NOT_ABS_LP);
        }
        return;
    }
    vised_dq.push_back(cur);
    mark_loop(out[cur]);
}

void solve(const string& origs, const string& tars) {
    // Original string -> target string.
    init();
    for (int i = 0; i < origs.size(); i++) {
        int och = ch2id(origs[i]);
        int tch = ch2id(tars[i]);
        if (out[och] && out[och] != tch) {
            // If the original string already has a character to transform,
            // but it is not the target string's character, this creates a many-to-one conflict.
            cout << -1 << '\n';
            return;
        }
        if (!out[och]) {
            out[och] = tch;
            in_nds[tch].insert(och);
            if (och != tch) 
                diff_chs++;
        }
    }
    for (int i = 1; i <= CHSZ; i++) {
        if (in_nds[i].size() == 1) 
            in1_cnt++; // Count nodes with in-degree 1.
    }
    for (int i = 1; i <= CHSZ; i++) {
        if (out[i] && lpid[i] == UNKNOWN) {
            // Mark a cycle.
            vised_dq.clear();
            memset(vised, 0, sizeof(vised));
            mark_loop(i);
        }
    }
    if (origs != tars && in1_cnt == CHSZ) {
        // Determine whether all characters are in cycles by counting nodes with in-degree 1.
        cout << -1 << '\n';
        return;
    }
    cout << diff_chs + abs_lp_cnt << '\n';
}

int main() {
    int t;
    cin >> t;
    while (t--) {
        string origs, tars;
        cin >> origs >> tars;
        solve(origs, tars);
    }
}

[Stanford CS144] Lab 4 Record

2023-01-30T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Implementing the TCP State-Transition Diagram

The main purpose of Lab 4 is to combine the receiver and sender from the previous labs into a complete TCP protocol stack. Therefore, it is important to become familiar with the TCP state-transition diagram.

Here is a TCP state-transition diagram:

Establishing a TCP Connection

As shown above, TCP has two ways to establish a connection. The first is an active connection, in which a SYN packet is sent to the peer. The second is a passive connection, in which a SYN+ACK packet is sent in response after receiving a SYN packet.

Active Connection

For an active connection, we need to implement connect():

void TCPConnection::connect() {
    _shutted = false;
    _sender.fill_window();
    send_sender_segs();
}

The shutted variable indicates whether the connection has been closed and is used later in active(). We previously implemented TCPSender::fill_window(), which records whether the connection has been established and automatically sends a SYN packet if it has not.

However, TCPSender::fill_window() only pushes the packets to be sent into its _segments_out queue. We need to put these packets into TCPConnection’s _segments_out, so that Sponge will send them using IP.

Therefore, one purpose of send_sender_segs() after fill_window() is to move the packets in _segments_out into TCPConnection’s _segments_out.

Of course, TCPSender does not know some header information when sending a packet, such as win and ackno. The former indicates how much data TCPReceiver can still receive, while the latter indicates how much data TCPReceiver has already received. Therefore, we also need to fill in this information in send_sender_segs().

One relatively tricky part is the range of win in the header. The definition in TCPHeader is:

1	uint16_t win = 0; //!< window size

This is an unsigned 16-bit integer. However, TCPReceiver::window_size() returns a 64-bit integer:

size_t TCPReceiver::window_size() const { 
    // Number of bytes that can still be received starting from ackno.
    return _capacity - _reassembler.stream_out().buffer_size();
    // window_size() + buffer_size() = capacity.
}

If we directly assign the result of window_size() to win, overflow may occur, so assign it as follows:

1	seg.header().win = min(_receiver.window_size(), (size_t)numeric_limits<uint16_t>::max());

Passive Connection

Looking again at the state diagram, if the current state is LISTEN, then after receiving a SYN and replying with SYN+ACK, the connection is established.

But how do we determine the LISTEN state? A convenient method is to use the TCPState class provided by Sponge.

It can determine not only the overall state of TCPConnection, but also the states of TCPSender and TCPReceiver separately.

Here, LISTEN is an overall state.

In segment_received(), we can determine whether to perform a passive connection as follows:

1	bool passive_connect = (state() == TCPState::State::LISTEN && seg.header().syn);

If a passive connection is needed, write:

// Passively establish a connection when in the listen state.
bool passive_connect = (state() == TCPState::State::LISTEN && seg.header().syn);
// For the receiver: LISTEN.
// For the sender: CLOSED.

_receiver.segment_received(seg);  
// Call segment_received first so that we know which ackno to send.
if (passive_connect) {
    connect();
    return;
}

At this point, the connection has been established successfully. For every newly arriving packet, we only need to call _receiver.segment_received() and _sender.ack_received() in segment_received() to update the information and maintain the connection (_sender.ack_received() lets the sender know what the peer received, so it can retransmit missing data).

Closing a TCP Connection

Compared with establishing a connection, closing one is more complicated, and a “perfect” close cannot always be guaranteed.

In computer networking, a famous thought experiment describing why TCP cannot close a connection perfectly is the Two Generals’ Problem. The description on Wikipedia^[1] is as follows:

Two armies led by different generals are preparing to attack a fortified city. The armies camp in two valleys near the city. Since another valley separates the two hills, the generals can communicate only by sending messengers through the valley. However, the valley is occupied by the city’s guards, who may capture any messenger passing through it.
Although the armies have agreed to attack simultaneously, they have not agreed on an attack time. To attack successfully, both armies must attack at the same time. If only one army attacks, it will be defeated, so the generals must agree on an attack time and ensure that the other general knows that they agree to the plan.

General A first sends a messenger to General B saying, “Attack at 9:00 on August 4.” After sending the messenger, General A does not know whether it successfully passed through enemy territory. Worried about becoming the only attacking army, General A may hesitate to attack as planned.
To remove this uncertainty, General B can send General A a confirmation saying, “I received your message and will attack at 9:00 on August 4,” but the messenger carrying the confirmation may also be captured. Worried that General A did not receive the confirmation and will retreat, General B may hesitate to attack as planned.
Sending another confirmation seems to solve the problem—General A can send a new messenger saying, “I received your confirmation to attack at 9:00 on August 4.” But General A’s new messenger may also be captured. Clearly, no number of confirmations satisfies the second condition: both parties must ensure that the other has agreed to the plan. The generals will always doubt whether the last messenger they sent successfully crossed enemy territory.

TCP closing has the same problem. After A sends a disconnection message, B can send an ACK packet to indicate that it received the message. However, B does not know whether A received the ACK, and therefore worries about whether A will close normally. A can of course send another ACK, but this falls into the Two Generals’ dilemma.

Repeatedly sending confirmations may seem to reduce errors, but the TCP protocol does not reply to an ACK packet (a packet containing no actual data, only an ACK) with another ACK, so we need another solution.

As with connection establishment, we can discuss active and passive closing separately.

Passive Close

Compared with active closing, passive closing is relatively simple, so let us discuss it first.

The only difference between passively and actively closing endpoints is the order in which they send FIN packets. An active closer sends a FIN packet after sending all TCP packets produced by its own outgoing byte stream.

Although one side has sent FIN at this point, this does not mean that the connection is closed, because the passive side may still have data to send. After it finishes sending, the passive side also sends FIN and enters the LAST_ACK state.

The sole purpose of this state is to wait for the other side to acknowledge FIN. If the active side does not acknowledge it, the passive side must keep sending FIN to ensure that the peer received it.

After receiving the ACK, it can close directly.

Active Close

When the outgoing byte stream has been completely sent, one endpoint sends FIN and enters FIN_WAIT_1. This indicates that one direction of the TCP connection has closed: the current endpoint only receives data and will not send new data. After the peer acknowledges the FIN, the endpoint changes to FIN_WAIT_2 and waits for the peer to finish sending its data.

After receiving and acknowledging the peer’s FIN, the endpoint enters TIME_WAIT. This state means:

The endpoint has completed reassembling the incoming byte stream, and the incoming byte stream is closed.
The outgoing byte stream has been completely sent and acknowledged.

Although, after entering TIME_WAIT, we cannot know whether the peer received our acknowledgment of its FIN, if it did not, it will most likely retransmit FIN within a certain time (TCP’s timeout retransmission mechanism is implemented by TCPSender).

Although the network may be congested, if we wait (linger) for a relatively long time and the peer does not retransmit, it is likely that the peer received the acknowledgment and closed the connection.

The assignment specifies this waiting time:

it has been at least 10 times the initial retransmission timeout (cfg.rt_timeout) since the local peer has received any segments from the remote peer.^[2]

With the default cfg.rt_timeout, the total waiting time is at least 10 seconds.

Implementing Passive Close

Earlier we mentioned that the passive side does not need to wait, or linger. It can be implemented as follows:

// In segment_received.

    // The endpoint that sends FIN later (receives FIN first) does not need to linger.
    // This is the transition from ESTABLISHED to CLOSE_WAIT.
    if (TCPState::state_summary(_receiver) == TCPReceiverStateSummary::FIN_RECV &&
        TCPState::state_summary(_sender) == TCPSenderStateSummary::SYN_ACKED) {
        // We cannot directly use state() == CLOSE_WAIT because CLOSE_WAIT also requires linger_after to be false,
        // while we assume lingering first.
        _linger_after_streams_finish = false;
    }
    
    // This is the transition from LAST_ACK to CLOSED.
    if (TCPState::state_summary(_receiver) == TCPReceiverStateSummary::FIN_RECV &&
        TCPState::state_summary(_sender) == TCPSenderStateSummary::FIN_ACKED && !_linger_after_streams_finish) {
        // We cannot use state() == LAST_ACK because that means the sender sent FIN, not that FIN was acknowledged (FIN_ACKED).
        _shutted = true;
        return;
    }

Implementing Active Close

Because _linger_after_streams_finish defaults to true, if it was not set to false in the preceding check, we are the active closer.

The only function in TCPConnection that can obtain the current time is tick(). To implement directly closing the connection after a timeout, add the following to tick():


if (state() == TCPState::State::TIME_WAIT && _since_lst_rx_ms >= 10 * _cfg.rt_timeout) {
        _shutted = true;
        _linger_after_streams_finish = false;
    }

After adding a great many details, the tests pass. (The assignment’s collaboration policy says that the code cannot be made public, so only some code snippets are included here.) The test result is:

1
2
3

./tcp_benchmark
CPU-limited throughput                : 0.37 Gbit/s
CPU-limited throughput with reordering: 0.36 Gbit/s

To be honest, the speed is still relatively slow. The main reason can also be seen from the earlier flame graph: string copying and processing. After optimizing it, I may write another blog post introducing the optimization process and content.

[Stanford CS144] Lab 0–Lab 3 Lab Notes

2022-12-25T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Note: Because both the lab guide and the course files^[1] explicitly state that code must not be published, the lab notes on this blog mainly record ideas and a few core code snippets; I will not publish the complete repository.

Lab 0: Networking Warmup

The lab requires implementing a reliable byte stream in memory (ByteStream), which feels rather similar to a Unix pipe. A first-in, first-out structure like this could be implemented very simply with the STL queue. However, since the lab requires a fixed-capacity byte stream, I personally think it is more appropriate to simulate it directly with an array, which should also be faster.

More specifically, use a string to store the data—I did not use a raw character array because the lab guide recommends modern C++ style and avoiding new for manual memory allocation—along with head and tail pointers that point to the beginning and end of the byte stream. This implements a circular queue. The peek_output() function is roughly:

string ByteStream::peek_output(const size_t len) const {
    size_t peek_size = min(buffer_size(), len);
    size_t i = 0;
    string ret = "";
    ret.resize(peek_size);
    while (i < peek_size) {
        ret[i] = _data[(_head + i) % _capa];
        i++;
    }
    return ret;
}

Although this implementation looks intuitive, its performance is relatively poor. The main reason is that the circular queue uses the modulo operation extensively, which causes a significant slowdown. Since I had not started Lab 4 at the time, I did not consider performance too deeply. The Lab 0 test results in release mode were:

[100%] Testing Lab 0...
Test project /mnt/e/ocourses/st_cs144/sponge/build
    Start 26: t_byte_stream_construction
1/9 Test #26: t_byte_stream_construction .......   Passed    0.01 sec
    Start 27: t_byte_stream_one_write
2/9 Test #27: t_byte_stream_one_write ..........   Passed    0.01 sec
    Start 28: t_byte_stream_two_writes
3/9 Test #28: t_byte_stream_two_writes .........   Passed    0.01 sec
    Start 29: t_byte_stream_capacity
4/9 Test #29: t_byte_stream_capacity ...........   Passed    0.22 sec
    Start 30: t_byte_stream_many_writes
5/9 Test #30: t_byte_stream_many_writes ........   Passed    0.01 sec
    Start 31: t_webget
6/9 Test #31: t_webget .........................   Passed    0.81 sec
    Start 53: t_address_dt
7/9 Test #53: t_address_dt .....................   Passed    0.05 sec
    Start 54: t_parser_dt
8/9 Test #54: t_parser_dt ......................   Passed    0.01 sec
    Start 55: t_socket_dt
9/9 Test #55: t_socket_dt ......................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 9

Total Test time (real) =   1.17 sec
[100%] Built target check_lab0

Lab 1: Stitching Substrings into a Byte Stream

This lab requires implementing a “reassembler”: it rearranges different data fragments into a continuous byte stream according to their supplied starting indices. We must also put received data into the byte stream as quickly as possible. In other words, if every character in $[0,i]$ has been received, that entire section should be inserted into the byte stream immediately.

Even before considering the experiment itself, the requirements in the lab guide are rather difficult to understand, especially the concept of capacity. In simple terms, it is the size of unread data in the byte stream plus the receive range of the reassembler.

Put another way, the reassembler has limited capacity. If the index of a data segment is too large, the reassembler can discard it directly. The more unread data there is in the byte stream, the smaller the minimum index that will be discarded. This means capacity is shared between bytes already assembled but not yet read and out-of-order bytes still waiting in the reassembler; treating those as two independent capacities would allow the implementation to retain more data than the lab permits.

Implementation

Rough Description

There are many ways to implement the reassembler. The simplest is to copy every arriving data segment, then insert data into the byte stream whenever a continuous run appears at the front of the reassembler.

This algorithm is obviously very inefficient. Every newly arriving data segment must be traversed completely, even if exactly the same data has already been received.

To avoid repeated copying, I implemented a data structure dedicated to maintaining a “set of segments.”

Any newly arriving data segment can be represented as an interval $[l,r)$ . We can also maintain a set of segments, denoted by $u$ , representing the ranges not yet received. For a new segment $x=[l,r)$ , if we can calculate the portions where $u$ and $x$ overlap—that is, $u\cap x$ —we need only traverse those portions: the unfilled segments covered by $x$ . If the length of $u\cap x$ is 0, the new data contains no previously unreceived portion, so we can return immediately and avoid the repeated traversal described above.

After writing the $u\cap x$ portion of the new segment, we must also change $u$ by removing $u\cap x$ , indicating that this portion has now been received. The data structure therefore records missing ranges rather than received ranges. A repeated segment has an empty intersection with the missing set and can be rejected without touching every byte again.

Example

The description above may not be very clear, so consider an example.

Suppose the goal is to receive a data segment covering $[0,10)$ . At the beginning, no data has arrived, so $u$ is the range $[0,10)$ .

Now a new segment $x=[2,5)$ arrives. We obtain $u\cap x=[2,5)$ , meaning that none of the data in $x$ is duplicated.

After filling $x$ , perform $u=u-(x\cap u)$ . The minus sign here does not denote a set difference in the formal sense; it means removing a portion from $u$ . This indicates that $x\cap u$ is no longer unfilled. The value of $u$ becomes $\{[0,2),[5,10)\}$ .

Now receive another segment $y=[1,6)$ . It completely covers the previous segment $x=[2,5)$ , but there is no need to traverse the already filled portion again. Instead, fill only the intersection $u\cap y=\{[1,2),[5,6)\}$ .

Requirements

At this point, the required data structure is relatively clear. We should implement two classes: Seg, representing one segment, and Segs, representing a set containing many segments.

Segs needs the following operations:

Find its intersection with a Seg, the $u\cap x$ operation described above.
Delete a Seg, the $u=u-(u\cap x)$ operation described above.

A Segs can contain many Seg objects. To calculate the intersection of a Segs object $a$ and a $Seg object $b$, first find a subset $c$ of $a$ in which every $Seg overlaps $b$ , approximately as follows:

1
2
3

         1        2(c1)            3(cn)            4
Segs a : |---|    |-----|          |--------|       |---|
Seg  b :        |-----------------------|

Segments 2 and 3 of $a$ overlap $b$ and therefore belong to subset $c$ .

Algorithm

Traversing the small Seg objects of $a$ one by one has linear complexity and is not much better than the naive algorithm.

The optimization I use is binary search.

Let the first segment of subset $c$ be $c_1$ —segment 2 in the diagram—and let its last segment be $c_n$ —segment 3 in the diagram.

Observation shows that $c_1$ must be the first segment whose right endpoint is greater than the left endpoint of $b$ . Likewise, $c_n$ must be the last segment whose left endpoint is less than the right endpoint of $b$ . Expressions of this “maximize a minimum” form can clearly be solved by binary search, provided that the multiple $Segobjects stored inSegs` are ordered.

Because the Segs class frequently inserts and deletes segments, I used std::set to store them while maintaining an ordered state for convenient queries.

This reduces the complexity of finding $c_1$ and $c_n$ to $\log(\text{number of segments})$ . Only the segments between these two iterators can overlap the query, so later intersection and deletion operations can work on the relevant consecutive range rather than scanning the entire set.

The function that queries $c_1$ and $c_n$ is the core of the entire data structure. It is shown below; the other portions are inconvenient to show because of the rule against publishing code.

template bool REC_LEN>
typename std::pair<typename Segs::s_iter_t, typename Segs::s_iter_t>
Segs::intersect_iter(const Seg &b) const {
    // return the first and last iterator of the intersected segments
    // Return iterators to the first and last segments that overlap b.
    if (b.len() == 0)
        return {_segs.end(), _segs.end()};
    auto fir = fir_GT_iter_r(b.l); // c1 above: the first segment whose right endpoint exceeds the query's left endpoint
    if (fir != _segs.end() && ((*fir) ^ b).len() == 0)  // if no intersection
        fir = _segs.end();
    auto las = lst_LT_iter(b.r); // cn above: the last segment whose left endpoint is below the query's right endpoint
    if (las != _segs.end() && ((*las) ^ b).len() == 0)
        las = _segs.end();

    // Handle cases in which c1 or cn was not found.
    if (fir == _segs.end() && las != _segs.end())
        fir = las;
    if (fir != _segs.end() && las == _segs.end())
        las = fir;
    return {fir, las};
}

Then, in StreamReassembler::push_substring, the data can be filled directly according to the ranges supplied by Segs:

……
    // insert new arrival into _tmp
    const Seg coverage{index, index + data.size()}; // Range of the newly arrived segment
    auto &&unfilled_intersect = _unfilled_segs ^ coverage; // ^ is overloaded here to mean intersection.
    for (auto &s : unfilled_intersect) {
        // s denotes an unfilled segment.
        for (size_t i = s.l; i < s.r && (i - _fir_unpushed_idx) <= _capacity; i++) {
            _tmp[i - _fir_unpushed_idx] = data[i - index];
            // _tmp[0] corresponds to _fir_unpushed_idx, the first position not yet inserted
            // into the byte stream. Add an offset, while (i - _fir_unpushed_idx) <= _capacity
            // ensures that _tmp does not go out of bounds.
            _unassembled_bt++;
        }
    }
    _unfilled_segs -= coverage;
    // find the first unfilled segment, before this segment, all data are filled
……

The performance of push_substring implemented this way is fairly satisfactory:

[100%] Testing the stream reassembler...
Test project /mnt/e/ocourses/st_cs144/sponge/build
      Start 18: t_strm_reassem_single
 1/16 Test #18: t_strm_reassem_single ............   Passed    0.01 sec
      Start 19: t_strm_reassem_seq
 2/16 Test #19: t_strm_reassem_seq ...............   Passed    0.01 sec
      Start 20: t_strm_reassem_dup
 3/16 Test #20: t_strm_reassem_dup ...............   Passed    0.01 sec
      Start 21: t_strm_reassem_holes
 4/16 Test #21: t_strm_reassem_holes .............   Passed    0.01 sec
      Start 22: t_strm_reassem_many
 5/16 Test #22: t_strm_reassem_many ..............   Passed    0.10 sec
      Start 23: t_strm_reassem_overlapping
 6/16 Test #23: t_strm_reassem_overlapping .......   Passed    0.01 sec
      Start 24: t_strm_reassem_win
 7/16 Test #24: t_strm_reassem_win ...............   Passed    0.10 sec
      Start 25: t_strm_reassem_cap
 8/16 Test #25: t_strm_reassem_cap ...............   Passed    0.07 sec
      Start 26: t_byte_stream_construction
 9/16 Test #26: t_byte_stream_construction .......   Passed    0.01 sec
      Start 27: t_byte_stream_one_write
10/16 Test #27: t_byte_stream_one_write ..........   Passed    0.01 sec
      Start 28: t_byte_stream_two_writes
11/16 Test #28: t_byte_stream_two_writes .........   Passed    0.01 sec
      Start 29: t_byte_stream_capacity
12/16 Test #29: t_byte_stream_capacity ...........   Passed    0.20 sec
      Start 30: t_byte_stream_many_writes
13/16 Test #30: t_byte_stream_many_writes ........   Passed    0.01 sec
      Start 53: t_address_dt
14/16 Test #53: t_address_dt .....................   Passed    0.05 sec
      Start 54: t_parser_dt
15/16 Test #54: t_parser_dt ......................   Passed    0.01 sec
      Start 55: t_socket_dt
16/16 Test #55: t_socket_dt ......................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 16

Total Test time (real) =   0.70 sec
[100%] Built target check_lab1

Later, I also used perf to generate flame graphs and tried to optimize the implementation further. The generated results are below. These SVG images are interactive, but they need to be opened in a separate window.

The first image is from debug mode, and the second is from release mode. In release mode, many functions are inlined, making analysis difficult. In debug mode, however, we can see that operations on Segs consume little time inside push_substring; instead, string operations on the deque are very expensive, such as:

1 2	_ZNSt5dequeIcSaIcEEixEm -> std::deque >::operator[](unsigned long) _ZNSt5dequeIcSaIcEE5frontEv -> std::deque >::front()

Clearly, using a deque to store temporary data is not a good choice. However, since Segs performs well, I will leave it unchanged for now and address the string-copying problem specifically when optimizing performance in Lab 4.

Lab 2: The TCP Receiver

This lab has two parts. The first requires implementing conversions between relative and absolute sequence numbers, while the second actually uses the wrapper class implemented earlier to write the TCP receiver.

Completing this lab requires a basic understanding of the TCP header. First, one message may be split into many small segments for transmission under TCP, and every segment has a header. The SYN and FIN flags mark the beginning and end of the transmission, respectively.

That is, if SYN in a header is true, the TCP packet is the first packet of the entire message. FIN similarly identifies the last packet.

We normally use 0 as the first index in a sequence of data, such as a character array. TCP does not do this: the index of the first data is randomized. Every TCP header contains a sequence number, seqno, that denotes the starting index of the data in that packet. A packet containing SYN is the first packet in the whole sequence, so its seqno is the first index of the whole sequence. We call this first index the ISN, or initial sequence number.

Why use a random sequence number? The main reason is to avoid confusion with historical data. During an earlier connection, some packets may have been transmitted extremely slowly because of network congestion and may arrive only after the connection has closed. If sequence numbers were not randomized, the historical packet’s sequence number would very likely fall inside the receive window of the new connection and be accepted incorrectly^[2].

Sequence-Number Wrapper

Although the TCP packet’s index is randomized, when we use it—for example, in the previously implemented push_substring function—we still need to convert it into a zero-based index. This index differs from seqno and is 64 bits wide.

The lab guide calls the zero-based index the absolute sequence number, or absolute seqno. We need to write a class specifically to convert between these two kinds of sequence number.

Converting an absolute sequence number to a wrapped seqno is simple: return ISN plus the absolute sequence number. Natural overflow directly produces the wrapped sequence number.

Converting a wrapped seqno back to an absolute sequence number is not as simple. A wrapped seqno is 32 bits, while the absolute sequence number is 64 bits, so the same wrapped value can correspond to multiple absolute values separated by multiples of $2^{32}$ . The required $unwrapfunction therefore receives an additionalcheckpoint`; the converted absolute sequence number must be the one closest to that checkpoint. Without the checkpoint, the low 32 bits alone cannot tell us which wraparound period contains the intended absolute position.

The problem is clearer in mathematical language. Let the checkpoint be $c$ , the wrapped sequence number be $s$ , and $M=2^{32}$ .

We need to find an absolute sequence number $s_a$ such that $s_a\equiv s-\text{isn}\pmod M$ while minimizing $|s_a-c|$ .

My implementation is below. It may look confusing at first glance; in fact, the explanation below is also rather confusing. I tried several ways to express the idea, but my mathematics and writing abilities prevented me from explaining it clearly.

//! \param n The relative sequence number
//! \param isn The initial sequence number
//! \param checkpoint A recent absolute 64-bit sequence number
uint64_t unwrap(WrappingInt32 n, WrappingInt32 isn, uint64_t checkpoint) {
    WrappingInt32 wrapped_ckp = wrap(checkpoint, isn); 
    // Reduce modulo 2^32 and add isn.
    // This converts an absolute checkpoint into a checkpoint relative to isn.
    int32_t offset = n - wrapped_ckp;
    static constexpr uint32_t MX32 = numeric_limits<uint32_t>::max();
    int64_t ret = offset + checkpoint;
    if (ret < 0)
        return ret + MX32 + 1; 
    return ret;
}

Here, offset is the distance, modulo $2^{32}$ , from $checkpoint + isnto theseqno` being converted. It may be positive or negative.

0     2^32     2*2^32     3*2^32
|        |        |        |
|--------|--------|--------|
 |     |                 |
seqno  ckp + isn       ckp + isn (actual)
 |<--->|
  offset

To obtain a seqno closest to checkpoint + isn, add the newly obtained offset to checkpoint + isn. This is equivalent to adding some multiple of $2^{32}$ to the $seqno`.

Subtracting isn from offset + checkpoint + isn yields the absolute sequence number, because a wrapped and absolute sequence number differ by exactly isn.

Therefore, the absolute sequence number equals offset + checkpoint.

However, this direct calculation may not produce the optimal solution. The following is the result of using the method directly:

0     2^32     2*2^32     3*2^32
|        |        |        |
|--------|--------|--------|
                   |     |               
                  seqno  ckp + isn      
                   |<--->|
                    offset

We can see that adding $2^{32}$ directly to the current $seqnowould place it closer tocheckpoint + isn`, while still satisfying $s_a\equiv s-\text{isn}\pmod M$ .

Such a failure to find the optimum can only occur when $|\text{offset}|>2^{32}\div2$ .

Adding any multiple of $2^{32}$ to $seqno` does not change it modulo $2^{32}$ . The offset does change, however, and our goal is to minimize that offset.

For example, if $\text{offset}=-2^{32}+1$ , which certainly satisfies the preceding inequality, then:

$(\text{offset}+2^{32})=1$

As in the preceding example, adding $2^{32}$ directly to $seqno` gives:

2^32     2*2^32   3*2^32   4*2^32
|        |        |        |
|--------|--------|--------|
                |  |                 
        ckp + isn  seqno       
                <-->
                offset

Natural overflow handles this problem for us, so we do not need to deal with it ourselves.

Notice that offset is stored as an int32_t. It is signed and has exactly the range $[-2^{32}\div2,2^{32}\div2-1]$ .

Therefore, whenever $|\text{offset}|>2^{32}\div2$ , offset automatically adds or subtracts a multiple of $2^{32}$ from itself to minimize its value.

This implementation still has a bug, however. Consider:

0                      2^32
|-----------------------|
  |                  |
  ckp+isn            seqno
  |<---------------->|
          offset

The offset here is clearly positive and greater than $2^{31}$ . Subtracting $2^{32}$ from $seqnowould reduce the absolute value of the offset, but it would also makeseqno` negative, which is clearly invalid. The following lines prevent a negative result: if the result is negative, add $2^{32}$ back.

static constexpr uint32_t MX32 = numeric_limits<uint32_t>::max();
    int64_t ret = offset + checkpoint;
    if (ret < 0)
        return ret + MX32 + 1;

CF1774C Solution

2022-12-17T22:33:18.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

To complain a little, the official solution was rather difficult to understand. I remained quite confused after reading it for a long time (actually, I am also just not good enough). After understanding it, I felt that this problem was quite clever, so I came to write a solution.

Approach

We first need to make an observation: for the string $s$ , its final consecutive segment does not increase the number of possible winners. For example, when $s = \texttt{0011}$ , the ending $\texttt{11}$ does not increase the number of possible winners.

Why? Suppose that, after any number of matches, the set of possible player combinations is $t$ . Then, for any $x \in t$ , after any number of consecutive matches in environment $1$ , the final winner must be the player in $x$ with the highest temperature. This is because every remaining player must play continuously in environment $1$ , and the only player who can win every match must be the largest. Similarly, after the players in $t$ play any number of consecutive matches in environment $0$ , the final winner must be the player in $x$ with the lowest temperature.

For example, when $s = \texttt{111}$ , player 4 must be the final winner.

Thus, if the ending segment is $1$ (the same applies to an ending of $0$ ; for convenience, the examples below use $1$ ), we only need to calculate how many player combinations with different maximum values (player temperatures) can be constructed by the preceding part. This tells us the answer for $s$ at its current length.

Now consider how to construct the maximum number of player combinations with different maximum values. If there are $n$ players, then before any matches, the maximum value is $n$ . To make the maximum value different, we can only remove the current maximum.

Special Case

The preceding description may be relatively abstract. The example $s = \texttt{0011}$ makes it easier to understand.

For the first $0$ , every player other than the one with the lowest temperature in the player combination can be removed (that lowest-temperature player can win no matter what). We can make the player with the lowest temperature play against any other player, giving the following cases:

$1 : \texttt{1234} \cancel{\texttt{5}} \\2 : \texttt{123} \cancel{\texttt{4}} \texttt{5} \\3 : \texttt{12} \cancel{\texttt{3}} \texttt{45} \\4 : \texttt{1} \cancel{\texttt{2}} \texttt{345} \\$

Observation shows that only the first case changes the maximum value (~~why? Because it removes the maximum~~). In the other cases, a consecutive segment of numbers at the end must be removed before the maximum value can change.

This is where the second $0$ has an effect. In the second case, it can remove $5$ , making the maximum value of the player combination become $3$ . Following this pattern, we can generalize the following conclusion: suppose the length of the consecutive segment before the ending segment is $l$ . The number of player combinations it can produce with different maximum values is $l + 1$ . Specifically, the possible range of maximum values is $[n - l, n]$ . (The $+1$ is because we can choose not to change the original maximum value.)

At this point, we can already find the answer when $s$ contains only two consecutive segments. It is $n - k$ , where $k = |s| - l$ represents the length of the ending segment.

Generalization

Now let us change the example to $s = \texttt{1011}$ and see whether the conclusion still holds ( $s$ contains more than one segment). Similarly, we can list the possible player combinations after the first match. Because the first environment is $1$ , only players other than the maximum can be removed:

$1 : \texttt{123} \cancel{\texttt{4}} \texttt{5} \\2 : \texttt{12} \cancel{\texttt{3}} \texttt{45} \\3 : \texttt{1} \cancel{\texttt{2}} \texttt{345} \\1 : \cancel{\texttt{1}} \texttt{2345} \\$

Although none of these cases changes the maximum value of the player combination, we only need to play one more match in the following $0$ environment and remove $5$ to produce two new maximum values. In the first case, the maximum becomes $3$ ; in the second case, it becomes $2$ . There are also $n - k$ possible answers in total.

What if there are even more segments? For example, $s = \texttt{01011}$ . The conclusion still holds. We can regard $\texttt{010}$ as a group of environments, in which $0$ can remove any player in the range $[2, n]$ , while $1$ can remove any player in $[1, n - 1]$ . Combining these two kinds of environments allows us to choose any three players from $[1, n]$ to remove, constructing four different maximum values (depending on how many of the players with the highest temperatures are removed from the beginning).

Code and Implementation

Through the preceding examples, we have determined that solving the problem only requires knowing the length of the final consecutive segment in the substring $[1, i]$ of $s$ . However, scanning it again for every $i$ is too slow, so we need to use something similar to dynamic programming. The specific explanation is in the code comments:

#include
using namespace std;
#define ll long long
int main(){
    int t;
    cin >> t;
    while(t--){
        int n;
        string s;
        cin >> n >> s;
        int cur0len = 0; // If the final consecutive segment consists of 0s, cur0len represents its length.
                         // Otherwise, cur0len is 0.
        int cur1len = 0; // The same applies to 1.
        int curn = 2;    // Initially there are two players.
        for (char ch : s){
            int x = ch - '0';
            if (x == 0){
                cur0len++;   // If the current character is 0, the consecutive segment ending in 0 is longer than before.
                cur1len = 0; // The final character of s is no longer 1.
            } else {
                cur1len++;
                cur0len = 0;
            }
            cout << curn - (x ? cur1len : cur0len) << " ";
            // n - k from the preceding discussion.
            curn++;
        }
        cout << '\n';
    }
}

Derivation and Implementation of a Dense Neural Network on MNIST

2022-10-31T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

upd@2022/11/5: Added the concrete implementation and corrected several notation errors in the derivation.
upd@2024/6/8: Corrected a subscript error in the derivation.

The main purpose of backpropagation is to calculate the partial derivatives of the neural-network error with respect to parameters such as biases and weights, allowing gradient descent to be performed. The first half of this article derives the algorithm; the second half uses a fully connected neural network and the MNIST dataset to implement handwritten-digit recognition.

This algorithm was still very difficult for me to understand, so I wrote these notes to prevent myself from forgetting it. Another reason is that neural-network formulas have so many superscripts and subscripts that it is very easy to write one incorrectly in a physical notebook. If you do not yet have a basic understanding of neural networks, I recommend 3Blue1Brown’s neural-network video series.

I must also say that MqCreaple is truly incredible: after watching the videos, he directly derived every formula by hand—~~even more astonishingly, he managed to teach someone like me~~.

Formula Derivation

Notation and Language Conventions

$\sigma()$ denotes the activation function.
$E$ denotes the final error.
$\operatorname{err}()$ denotes the error function.
$\hat y$ denotes the neural network’s prediction, while $y$ denotes the actual answer.
$l$ denotes a neural-network layer; a smaller value means the layer is closer to the input layer.
$w^{[l]}_{ji}$ denotes an edge from node $j$ in layer $l$ to node $i$ in layer $l-1$ .
$b^{[l]}_i$ denotes the bias of node $i$ in layer $l$ .
$z^{[l]}_i$ denotes the output of node $i$ in layer $l$ before applying the activation function.
$a^{[l]}_i$ denotes $\sigma(z^{[l]}_i)$ , the node’s output after applying the activation function.
$n^l$ denotes the number of nodes in layer $l$ .
An underline beneath a variable indicates that it is treated as a constant, such as $\underline{x}$ .
A layer “before” another layer has a smaller $l$ , and a layer “after” it has a larger $l$ .

A Neural Network with One Node per Layer

First consider the simplest fully connected neural network, in which every layer has only one node. The following diagram represents the process of calculating a single node’s output $a^{[l]}$ . The variable at the starting point of an arrow and the corresponding function produce the variable at the arrow’s endpoint.

    graph TB alm1["a(l-1)"] & w & b --> z --> al["a(l)"] --> Error y-->Error

Written as functions, the process is:

$z^{[l]} = wa^{[l-1]} + b \\a^{[l]} = \sigma(z^{[l]}) \\E = \operatorname{err}(a^{[l]}, y)$

To perform gradient descent on the weight $w^{[l]}$ according to the error, we need the partial derivative of the error with respect to the weight:

$\frac{\partial E}{\partial w^{[l]}}$

We use a partial derivative because the calculation of $z^{[l]}$ depends on three variables, while we want to know how changing $w^{[l]}$ affects the error.

When taking the partial derivative, assume that the other variables are constants. Only one variable changes, along with the variables directly affected by it—in this case, the chain $w^{[l]}\to z^{[l]}\to a^{[l]}\to E$ . Constants are underlined in the following expression:

$E = \operatorname{err}(\sigma(w^{[l]}\underline{a^{[l-1]}} + \underline{b}), \underline{y})$

We can now apply the chain rule:

$\begin{align*} E^\prime = \operatorname{err}^\prime(&\sigma(w^{[l]}\underline{a^{[l-1]}} + \underline{b}), \underline{y}) \\ \cdot &\sigma^{\prime}(w^{[l]}\underline{a^{[l-1]}} + \underline{b}) \\ \cdot &(w^{[l]}\underline{a^{[l-1]}} + \underline{b})^\prime\end{align*}$

In another form, which will be more convenient later:

$\frac{\partial E}{\partial w^{[l]}} = \frac{\partial z^{[l]}}{\partial w^{[l]}} \cdot \frac{\partial a^{[l]}}{\partial z^{[l]}} \cdot \frac{\partial E}{\partial a^{[l]}}$

The intermediate partial derivatives in the chain rule can then be calculated. Assuming the error function is squared error, the expression becomes:

$\begin{align*} \frac{\partial z^{[l]}}{\partial w^{[l]}} &= (w^{[l]}\underline{a^{[l-1]}} + \underline{b})^\prime = a^{[l-1]} \\ \frac{\partial a^{[l]}}{\partial z^{[l]}} &= \sigma^{\prime}(z^{[l]}) \\ \frac{\partial E}{\partial a^{[l]}} &= 2(a^{[l]} - y)\end{align*}$

Be careful not to reverse $2(a^{[l]}-y)$ . For example, when $a^{[l]}$ is too large, we also want the derivative to be large so that the value being adjusted can have the derivative subtracted from it.

The preceding calculation gives the error’s partial derivative with respect to the weight. For the bias and the previous layer’s output, simply replace $\frac{\partial z^{[l]}}{\partial w^{[l]}}$ in the formula. In other words, let the previous layer’s output or this layer’s bias affect $z^{[l]}$ instead of the weight.

For $b^{[l]}$ , use:

$\frac{\partial z^{[l]}}{\partial b^{[l]}} = (\underline{w^{[l]}a^{[l-1]}} + b)^\prime = 1$

For $a^{[l-1]}$ , use:

$\frac{\partial z^{[l]}}{\partial a^{[l-1]}} = (\underline{w^{[l]}}a^{[l-1]} + \underline{b})^\prime = w^{[l]}$

Now consider the following network:

    graph TB al["a(l)"] & wlp["w(l + 1)"] & blp["b(l + 1)"] --> zlp["z(l + 1)"] --> alp["a(l + 1)"] --> ...many other layers __["w(l + 2)"] & _["b(l + 2)"]-->...many other layers

Here, the layer after $a^{[l]}$ does not connect directly to the error function; many layers intervene. We can no longer calculate $\frac{\partial E}{\partial a^{[l]}}$ directly, and therefore cannot directly calculate the partial derivatives for $w^{[l]}$ and $b^{[l]}$ , because $E$ lies many layers later. This is where backpropagation is required.

We know:

$\frac{\partial E}{\partial a^{[l]}} = \frac{\partial z^{[l+1]}}{\partial a^{[l]}} \cdot \frac{\partial a^{[l+1]}}{\partial z^{[l+1]}} \cdot \frac{\partial E}{\partial a^{[l+1]}}$

The expression shows that $\frac{\partial E}{\partial a^{[l]}}$ for an earlier layer can be derived from a later layer. Before calculating partial derivatives for weights and biases, start from the output layer and propagate $\frac{\partial E}{\partial a^{[l]}}$ backward one layer at a time.

Multiple Nodes per Layer

The backpropagation process in the preceding example is clear and involves no linear algebra. In a real neural network, however, every layer contains multiple nodes:

    graph LR     l1["(l-1)1"] & l2["(l-1)2"] & l3["(l-1)3"] ---> lp1["l1"] & lp2["l2"] & lp3["l3"]

Partial Derivative of the Error with Respect to a Weight

$w^{[l]}_{ji}$ denotes an edge connecting node $j$ in layer $l$ to node $i$ in layer $l-1$ . How do we calculate $\frac{\partial E}{\partial w^{[l]}_{ji}}$ ?

We can still use the original formula, because a layer with multiple nodes is fundamentally composed of multiple single-node layers. We must, however, pay attention to the subscripts:

$\begin{align*} \frac{\partial E}{\partial w^{[l]}_{ji}} &= \frac{\partial z^{[l]}_j}{\partial w^{[l]}_{ji}} \cdot \frac{\partial a^{[l]}_j}{\partial z^{[l]}_j} \cdot \frac{\partial E}{\partial a^{[l]}_j} \\ &= a^{[l-1]}_i \cdot \sigma^\prime(z^{[l]}_j) \cdot \frac{\partial E}{\partial a^{[l]}_j}\end{align*}$

Every variable related to layer $l-1$ uses subscript $i$ , such as $a^{[l-1]}_i$ . Intuitively, when a unit of weight changes, a larger input from the previous layer has a larger effect on the final error function. Variables related to layer $l$ use subscript $j$ .

Because $\sigma^\prime(z^{[l]}_j)\frac{\partial E}{\partial a^{[l]}_j}$ uses the same subscript throughout, call it $r_j$ to make the matrix-operation formula easier to write.

Rewriting the formula gives:

$\frac{\partial E}{\partial w^{[l]}_{ji}} = r_j a^{[l-1]}_i$

In the matrix form of $w^{[l]}$ , $j$ increases by row and $i$ increases by column. The derivative matrix is:

$\frac{\partial E}{\partial w^{[l]}} =\begin{bmatrix} r_1 a^{[l-1]}_1 & r_1 a^{[l-1]}_2 & \cdots & r_1 a^{[l-1]}_{n^{l-1}} \\ r_2 a^{[l-1]}_1 & r_2 a^{[l-1]}_2 & \cdots & r_2 a^{[l-1]}_{n^{l-1}} \\ \vdots & \vdots & \ddots & \vdots \\ r_{n^l}a^{[l-1]}_1 & r_{n^l}a^{[l-1]}_2 & \cdots & r_{n^l}a^{[l-1]}_{n^{l-1}}\end{bmatrix}$

This matrix is equal to:

$\begin{bmatrix} r_1\\r_2\\\vdots\\r_{n^l}\end{bmatrix}\cdot\begin{bmatrix} a^{[l-1]}_1 & a^{[l-1]}_2 & \cdots & a^{[l-1]}_{n^{l-1}}\end{bmatrix}$

We can therefore use a matrix-operation library such as NumPy to accelerate the calculation.

Partial Derivative of the Error with Respect to a Bias

This is relatively simple because $\frac{\partial z^{[l]}_j}{\partial b^{[l]}_j}=1$ , as shown earlier:

$\begin{align*} \frac{\partial E}{\partial b^{[l]}_j} &= \frac{\partial z^{[l]}_j}{\partial b^{[l]}_j}\cdot\frac{\partial a^{[l]}_j}{\partial z^{[l]}_j}\cdot\frac{\partial E}{\partial a^{[l]}_j}\\ &=1\cdot\sigma^\prime(z^{[l]}_j)\cdot\frac{\partial E}{\partial a^{[l]}_j}\\ &=r_j\end{align*}$

The error derivative with respect to the bias equals the $r_j$ used above. Implementations therefore generally calculate this first, then substitute $r_j$ into the weight formula.

Partial Derivative of the Error with Respect to the Previous Layer’s Input

Look again at the multi-node network, now focusing on the effect of a single node in layer $l-1$ on the later nodes:

    graph LR     l1["(l-1)1"]  ===> lp1["l1"] & lp2["l2"] & lp3["l3"]

$a^{[l-1]}_i$ can affect every $z^{[l]}_j$ . If layer $l$ is regarded as a function that receives $n^{l-1}$ values $a^{[l-1]}_i$ and outputs $n^l$ values $z^{[l]}_j$ , then every input variable is changing. The result is not a partial differential but a total derivative^[1].

By the definition of a total derivative, add the partial derivative with respect to every parameter. In this example:

$\frac{\partial E}{\partial a^{[l]}_i}=\sum_{j=1}^{n^{l+1}}\left(\frac{\partial z^{[l+1]}_j}{\partial a^{[l]}_i}\cdot\frac{\partial a^{[l+1]}_j}{\partial z^{[l+1]}_j}\cdot\frac{\partial E}{\partial a^{[l+1]}_j}\right)$

The factor $\frac{\partial z^{[l+1]}_j}{\partial a^{[l]}_i}$ needs careful handling. Remember that $w^{[l+1]}_{ji}$ is the edge connecting node $j$ in layer $l+1$ and node $i$ in layer $l$ .

Since

$z^{[l+1]}_j=w^{[l+1]}_{ji}a^{[l]}_i+b^{[l+1]}_j,$

we obtain

$\frac{\partial z^{[l+1]}_j}{\partial a^{[l]}_i}=w^{[l+1]}_{ji}.$

The factor $\frac{\partial a^{[l+1]}_j}{\partial z^{[l+1]}_j}\frac{\partial E}{\partial a^{[l+1]}_j}$ was explained earlier: it equals $r_j$ , the error derivative with respect to the bias.

Rewriting the complete expression gives:

$\frac{\partial E}{\partial a^{[l]}_i}=\sum_{j=1}^{n^{l+1}}\left(r_jw^{[l+1]}_{ji}\right).$

Now consider how to obtain $\frac{\partial E}{\partial a^{[l]}_j}$ with matrix operations. One workable method multiplies $r_j$ and $w^{[l+1]}_{ji}$ .

We accumulate over subscript $j$ . If $w^{[l+1]}$ is placed on the left, its $j$ coordinate must increase with the column number; in matrix multiplication $A\times B$ , rows of $A$ are dotted with columns of $B$ . If $r$ is placed on the right, its $j$ subscript must increase with the row number.

Because $j$ in $w^{[l+1]}$ originally increases by row, transpose it. The final result is:

$\frac{\partial E}{\partial a^{[l]}}=(w^{[l+1]})^T\times r,$

where $r$ is a column vector.

Implementation

This section uses the backpropagation algorithm derived above to implement a simple fully connected neural network, then uses that network to recognize handwritten digits in the MNIST dataset.

Data Preprocessing

To be honest, the MNIST dataset is rather troublesome. It uses a binary storage format, so reading its contents takes some work.

The code is as follows

The struct package may look confusing. It is simply a class specialized for processing binary data.

1	struct.unpack('>II', lfile.read(8))

This statement reads two four-byte unsigned integers in big-endian byte order from lfile. In >II, > indicates that the file uses big-endian byte order, while I indicates a four-byte unsigned integer.

The following np.fromfile serves a similar purpose, directly converting the binary file into a NumPy array. No byte order is specified, perhaps because NumPy uses big-endian by default.

Individual pixels in MNIST are integers in $[0,255]$ . We want floating-point values in $[0,1]$ , so divide by 255 before returning.

Keeping images in $[0,1]$ is important because passing a relatively large number to $sigmoid` can cause overflow. Although each layer’s initial weights are randomly generated between -1 and 1, a layer can still sometimes output a large value. Sigmoid is defined as:

$\sigma(x)=\frac{1}{1+e^{-x}}.$

If $x$ is too small, $e^{-x}$ becomes extremely large. NumPy performs its calculations through C and does not have Python’s built-in arbitrary precision, so such a value naturally causes overflow.

The final preprocessing step converts labels into one-hot form, making it convenient to calculate the gradient of the error over the complete network. np.eye(x) generates an $x\times x$ identity matrix, so $np.eye(10)[labels[i]]is naturally the one-hot encoding corresponding tolabels[i]`.

The `layer` Class

A single neural-network layer is fundamentally a function that receives a vector and outputs a vector. The function depends on many variables, including weights and biases, so we want a class that stores them.

Backpropagation also needs the variables stored in the class. It is best to implement a function that receives the derivative of the error with respect to the current layer’s output, along with other necessary data, and returns the derivative of the error with respect to the previous layer’s output.

Finally, the layer needs an interface for updating its weights and biases. If other layer types use data other than weights and biases, an abstract class can represent the data of different layers.

These requirements give the following abstract layer class. Every parameter name follows the mathematical formulas above; refer to the derivation if any are unclear.

# Location in the project: ./src/layer.py
from typing import *
import numpy as np
from nptyping import NDArray, Shape, Float
from . import util

class abs_layer():
    def __init__(self, insize: int, outsize: int, activ: util.Dfunc = util.sigmoid):
        self.insize = insize
        self.outsize = outsize
        self.activ = activ
    def get_z(self, ipt: NDArray) -> NDArray:
        """ 
            Return an output before the activation function is applied to the input.
        """
        pass
    def get_a(self, ipt: NDArray) -> NDArray:
        """ 
            Return an output after the activation function is applied to the input.
        """

        pass
    def get_derivatives(self, prev_a : NDArray, DE_over_cur_a: NDArray, cur_z: NDArray) -> List[NDArray]:
        """ 
            prev_a        : output of the preceding layer after its activation function
            DE_over_cur_a : derivative of the error with respect to the current layer's output
            cur_z         : current layer's output before its activation function
        """
        pass
    def descent(self, w, b):    
        """ 
            w : weight gradient
            b : bias gradient
        """
        pass

Here, util.Dfunc represents a differentiable function and is defined as follows:

# Location in the project: ./src/util.py
class Dfunc():
    """ 
        Represent a differentiable function: f is the original function and df is its derivative.
        If f is multivariable, df should return a vector of partial derivatives for its inputs.
    """

    def __init__(self, func: Callable, Dfunc: Callable):
        self.f = func
        self.Df = Dfunc

sigmoid = Dfunc(lambda x: 1 / (1 + np.exp(-x)),
                lambda x: np.exp(-x) / ((1 + np.exp(-x)) ** 2))

sq_err = Dfunc(lambda label, predict: np.sum((predict - label) ** 2),
               lambda label, predict: 2 * (predict - label))

A fully connected layer can be implemented as follows:

# Location in the project: ./src/layer.py
class dense_layer(abs_layer):
    def __init__(self, insize: int, outsize: int, activ: util.Dfunc = util.sigmoid) -> None:
        super(dense_layer, self).__init__(insize, outsize)
        self.wts = np.random.rand(outsize, insize) * 2 - 1
        self.bias = np.random.rand(outsize) * 2 - 1

    def get_z(self, ipt: NDArray) -> NDArray:
        return np.matmul(self.wts, ipt.reshape(ipt.size, 1)).reshape(self.outsize) + self.bias

    def get_a(self, ipt: NDArray) -> NDArray:
        return self.activ(self.get_z(ipt))

    def get_derivatives(self, prev_a : NDArray,  DE_over_cur_a: NDArray, cur_z: NDArray) -> List[NDArray]:
        if (DE_over_cur_a.size != self.outsize):
            raise Exception("size of DE_over_cur_a ({}) doesn't equal to number of node in this layer ({})".format(DE_over_cur_a.size, self.outsize),
                            DE_over_cur_a
                            )
        Dbias : NDArray = DE_over_cur_a * self.activ.Df(cur_z)
        DE_over_prev_a: NDArray = np.matmul(self.wts.T, Dbias)
        Dweight = np.matmul(
            Dbias.reshape(Dbias.size, 1),
            prev_a.reshape(1, prev_a.size)
        )
        
        return [DE_over_prev_a, Dweight, Dbias]
        # Return three variables: partial derivatives of the error with respect to the preceding
        # layer's output and the current layer's weights and biases.
    def descent(self, w : NDArray, b : NDArray) -> None:
        self.wts -= w
        self.bias -= b

Except for get_derivatives, the functions are relatively easy to understand. The following roughly explains that function.

The formula for the error derivative with respect to the preceding layer is:

$\frac{\partial E}{\partial a^{[l-1]}_j}=(w^{[l]})^T\times r.$

In the implementation, this corresponds to:

1	DE_over_prev_a: NDArray = np.matmul(self.wts.T, Dbias)

Here, Dbias equals $r$ :

$r_j=\sigma^\prime(z^{[l]}_j)\frac{\partial E}{\partial a^{[l]}_j}.$

This corresponds to:

1	Dbias : NDArray = DE_over_cur_a * self.activ.Df(cur_z)

The formula for the derivative of the error with respect to weights is:

$\begin{bmatrix}r_1\\r_2\\\vdots\\r_{n^l}\end{bmatrix}\cdot\begin{bmatrix}a^{[l-1]}_1&a^{[l-1]}_2&\cdots&a^{[l-1]}_{n^{l-1}}\end{bmatrix}.$

It corresponds to:

Dweight = np.matmul(
            Dbias.reshape(Dbias.size, 1),
            prev_a.reshape(1, prev_a.size)
        )

The `neu_net` Class

The network class connects different layers, passing the output of one layer as the input to the next. It can also backpropagate starting from the error function.

Initialization Function

# Location in the project: ./src/net.py

def __init__(self,  layer_sizes: List[int] | None = None, layers: List[layer.abs_layer] | None = None) -> None:
    """ 
        layer_sizes: the first value is the input size and the last is the output size
    """
    if (layers != None and layer_sizes != None):
        raise Exception(
            "should only provide either layer_sizes or layers",
            self
        )
    if (layers == None):
        layers: List[layer.abs_layer] = []
        for i in range(0, len(layer_sizes) - 1):
            # This layer's input equals the preceding layer's output and the next layer's input.
            layers.append(layer.dense_layer(
                insize=layer_sizes[i], outsize=layer_sizes[i + 1]))
    self.lays = layers
    self.num_lay = len(layers)
    self.err = util.sq_err
    for i in range(1, self.num_lay):
        if (self.lays[i - 1].outsize != self.lays[i].insize):
            raise Exception(
                "layer {}'s output ({}) not equal to layer {}'s input ({})".format(i-1, self.lays[i-1].outsize, i, self.lays[i].insize), self.lays)

There are two ways to initialize the network. Individual layer objects can be supplied directly and combined by the network class, or a list containing the node count of each layer can be supplied so that the initializer automatically creates the corresponding fully connected network.

Output Function

def get_predict(self, ipt : NDArray):
    lay_z: List[NDArray] = []
    lay_a: List[NDArray] = []
    lay_z.append(self.lays[0].get_z(ipt))
    lay_a.append(self.lays[0].activ.f(lay_z[0]))
    for i in range(1, self.num_lay):
        lay_z.append(self.lays[i].get_z(lay_a[i - 1]))
        lay_a.append(self.lays[i].activ.f(lay_z[i]))
    return [lay_z, lay_a]

def get_simple_predict(self, ipt : NDArray):
    return self.get_predict(ipt)[1][-1]

The neural network’s first layer is special: it does not connect to the output of a preceding layer, but directly uses ipt, so it must be handled separately.

Backpropagation

def bp(self, ipt: NDArray, label: NDArray, lrate: float):
    lay_z, lay_a = self.get_predict(ipt)                       # Output of every layer
    lay_Dw: List[NDArray] = [np.zeros(0)] * (self.num_lay)     # Derivatives with respect to weights
    lay_Db: List[NDArray] = [np.zeros(0)] * (self.num_lay)     # Derivatives with respect to biases
    DE_over_a: List[NDArray] = [np.zeros(0)] * (self.num_lay)  # Error derivatives w.r.t. node outputs
    DE_over_a[-1] = self.err.Df(label, lay_a[-1])

    for i in reversed(range(1, self.num_lay)):
        DE_over_a[i - 1], lay_Dw[i], lay_Db[i] = self.lays[i].get_derivatives(
            prev_a=lay_a[i - 1],
            DE_over_cur_a=DE_over_a[i],
            cur_z=lay_z[i]
        )

    lay_Db[0] = self.lays[0].activ.Df(lay_z[0]) * DE_over_a[0]
    lay_Dw[0] = np.matmul(
        lay_Db[0].reshape(lay_Db[0].size, 1),
        ipt.reshape(1, ipt.size)
    )
    for Dw, Db, lay in zip(lay_Dw, lay_Db, self.lays):
        lay.descent(Dw * lrate, Db * lrate)

The main purpose here is to call get_derivatives on every layer and obtain the derivatives with respect to the outputs, weights, and biases of different layers.

There are two special cases. First, the error derivative with respect to the final layer must be obtained from the error function and label:

1	DE_over_a[-1] = self.err.Df(label, lay_a[-1])

The error derivatives with respect to the first layer’s weights and biases can only be obtained from the input image:

lay_Db[0] = self.lays[0].activ.Df(lay_z[0]) * DE_over_a[0]
        lay_Dw[0] = np.matmul(
            lay_Db[0].reshape(lay_Db[0].size, 1),
            ipt.reshape(1, ipt.size)
        )

Results

The accuracy reaches 96%, which is quite good; training took a little over a minute. Of course, the training method still has substantial room for optimization, and I did not tune the parameters very much.

1.https://zh.wikipedia.org/wiki/全微分 ↩

# Location in the project: ./src/util
def load_mnist(path: str, pref: str = "train"):
    """ 
        path: dataset path
        data_type: dataset-name prefix (train or t10k)
    """
    label_path = os.path.join(path, "{}-labels.idx1-ubyte".format(pref))
    img_path = os.path.join(path, "{}-images.idx3-ubyte".format(pref))
    with open(label_path, 'rb') as lfile: # rb means read binary.
        magic, n = struct.unpack('>II', lfile.read(8))
        labels = np.fromfile(lfile, dtype=np.uint8)
    with open(img_path, 'rb') as ifile: # ifile means image file.
        magic, num, rows, cols = struct.unpack('>IIII', ifile.read(16))
        images = np.fromfile(ifile, dtype=np.uint8).reshape(
            len(labels), 28 * 28)
    label_one_hot = np.zeros((len(labels), 10), dtype=int)
    for i in range(len(labels)):
        label_one_hot[i] = np.eye(10)[labels[i]]
    return label_one_hot, images / 255.0

↩

2.Adapted from https://zhuanlan.zhihu.com/p/120378080 ↩

Ray Tracing: The Next Week Study Notes (1)

2022-10-20T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

After more than a month of intermittent work, I finally finished the material in the second book. As with the previous two articles, this article records some parts that personally took me a relatively long time to understand, as well as some new features I added on top of the original book.

For features already present in the original book, I use the book’s code directly; for features I added, I use my own code. Because my code differs substantially from the original book—even for features that were already there—a single code excerpt may be hard to understand. You can refer to my GitHub repository here: https://github.com/ttzytt/RTOW

`bvh_node`

This section is mainly about a few small details that I did not fully understand at the time. First is this:

bvh_node::bvh_node(
    std::vector>& src_objects,
    size_t start, size_t end, double time0, double time1
)

The first issue is the range handled by this constructor. Each subtree is not responsible for the hittable at position end. In other words, this constructor handles an interval of the form [start, end).

This also explains how sort is used in the code:

1	std::sort(objects.begin() + start, objects.begin() + end, comparator);

The interval actually sorted by std::sort() is of the form [) (~~I somehow never noticed this before~~). Therefore, objects.begin() + end does not include end here.

When sorting containers such as a vector, the usual method is sort(vec.begin(), vec.end()). At first glance, this appears not to include the element at .end(), but .end() actually points to an empty position—or, put another way, the position after the last element (~~I had not noticed this before either~~). Consequently, this form sorts the entire vector.

Spherical Texture Coordinates

class sphere : public hittable {
    ...
    private:
        static void get_sphere_uv(const point3& p, double& u, double& v) {
            // p: a given point on the sphere of radius one, centered at the origin.
            // u: returned value [0,1] of angle around the Y axis from X=-1.
            // v: returned value [0,1] of angle from Y=-1 to Y=+1.
            //     <1 0 0> yields <0.50 0.50>       <-1  0  0> yields <0.00 0.50>
            //     <0 1 0> yields <0.50 1.00>       < 0 -1  0> yields <0.50 0.00>
            //     <0 0 1> yields <0.25 0.50>       < 0  0 -1> yields <0.75 0.50>

            auto theta = acos(-p.y());
            auto phi = atan2(-p.z(), p.x()) + pi;

            u = phi / (2*pi);
            v = theta / pi;
        }
};

This code uses the atan2 function rather than the ordinary atan function. We know that the trigonometric function tan returns the slope of the tangent for the corresponding angle on a circle. Accordingly, atan returns the angle corresponding to a slope. When calculating texture coordinates, however, what we actually want is to obtain the corresponding angle from a coordinate on the circle. Of course, we could directly use atan(y/x), first calculating the slope and then the angle.

The problem is that a circle is described by an equation rather than a function: one x-coordinate can correspond to multiple y-coordinates. A single slope can therefore correspond to multiple angles. More specifically, although $(x, y)$ and $(-x, -y)$ correspond to different angles, their slopes are identical. If we used atan, we would also have to check the signs of the coordinates ourselves; atan2 effectively performs this work for us.

Checkerboard Texture

Implementation in the Book

virtual color value(double u, double v, const point3& p) const override {
    auto sines = sin(10*p.x())*sin(10*p.y())*sin(10*p.z());
    if (sines < 0)
        return odd->value(u, v, p);
    else
        return even->value(u, v, p);
}

This code multiplies together the values of the three components after multiplying them by $\pi$ . If the result is positive, it returns one color; otherwise, it returns the other. This may be difficult to understand at first glance, but it becomes much clearer if we first draw a two-dimensional version:

After adding another axis, the sign of $\sin()$ changes periodically, so we can see distinct layers. The colors flip between layers, while within a single layer the sign remains unchanged, so it can be treated directly as the two-dimensional version above:

To be honest, however, trigonometric functions are not the only periodic functions. The book writes it this way only to obtain a positive or negative sign rather than a specific value, so using $\sin$ really wastes some computing resources.

One very simple example would be to take $x$ modulo $100$ : return a positive number if the result is less than $50$ , and a negative number otherwise. For a more concise form, it could also be written as follows:

$y=\operatorname{mod}\left(x,100\right)-50$

Checkerboard Only on the Surface

It is not difficult to see that the checkerboard in the book is based on a point’s absolute coordinates in space. That is why the layering shown above appears. Since we can already calculate texture coordinates on a sphere (the book also discusses texture coordinates for other hittable objects, such as rectangular patches), we can instead create a checkerboard texture based on the object’s surface, as follows:

class surface_checker : public texture {
   public:
    using text_array = std::vector>;
    surface_checker() = default;
    surface_checker(const text_array& _texts,
            const std::pair _siz = {514, 114})
        : texts(_texts), polar_azim_siz(_siz) {}

    virtual color value(f8 polar, f8 azim, const pt3& p) const override {
        int x_idx = (i8)(azim * polar_azim_siz.first);
        int y_idx = (i8)(polar * polar_azim_siz.second / 2.0); 
        // The polar angle spans only a hemisphere. To obtain polar_azim_siz.second
        // cells vertically over the whole sphere, divide by two first.
        return texts[(x_idx + y_idx) % texts.size()]->value(polar, azim, p);
    }

    text_array texts;
    std::pair polar_azim_siz;  // Number of cells vertically and horizontally
};

Here, text_array allows the checkerboard to contain more than two colors, while (azim * polar_azim_siz.first) expands the original texture-coordinate range of $[0, 1]$ to polar_azim_siz.first, ensuring that the sphere has polar_azim_siz color changes. This produces the following result:

The code that generates this scene is as follows:

scene surf_check_sc() {
    hittable_list world;

    auto checker1 = make_shared(
        surface_checker::text_array{
            make_shared(color(0.2, 0.3, 0.1)),
            make_shared(color(0.9, 0.9, 0.9)),
            make_shared(color(0.3, 0.2, 0.15)),
            make_shared(color(0.15, 0.3, 0.9))},
        std::pair{60, 60});

    auto checker2 = make_shared(
        surface_checker::text_array{
            make_shared(color(0.2, 0.3, 0.1)),
            make_shared(color(0.9, 0.9, 0.9)),
        },
        std::pair{30, 30});

    world.add(make_shared(pt3(0, -10, 0), 10,
                                  make_shared(checker1)));
    world.add(make_shared(pt3(0, 10, 0), 10,
                                  make_shared(checker2)));

    f8 asp_ratio = 1.0;
    vec3 lookfrom = pt3(13, 2, 3) * 2;
    vec3 lookat = pt3(0, 0, 0);
    f8 vfov = 40.0;
    auto dist_to_focus = 10.0;
    auto aperture = 0;
    vec3 vup(0, 1, 0);
    auto cam_ptr = make_shared(lookfrom, lookat, vup, vfov, asp_ratio,
                                       aperture, dist_to_focus, aperture, 1.0);

    return scene(make_shared(world), blue_sky_back_ptr, cam_ptr);
}

Perlin Noise

Perlin noise is one of the more difficult points in the book to understand, but Perlin noise is based on ordinary value noise. Value noise simply generates random numbers at integer coordinates in space and then uses those integer-coordinate values to linearly interpolate values at other coordinates (if you do not understand linear interpolation, see this link; I personally think it explains the idea very clearly).

It works roughly as follows

Random numbers are generated at the intersections of the vertical and horizontal lines—the integer coordinates. The value at point p in the figure is linearly interpolated from the four surrounding key points (that is, points with integer coordinates, which generate random numbers). Ultimately, the value at p depends on its distances from those four key points and the random values at the four points.

The following is an example of two-dimensional value noise:

The generating code is as follows:

import numpy as np
import matplotlib.pyplot as plt
from math import *

XLEN = 25 # Number of integer points to generate
YLEN = 25
DIFF = 0.05

ptsx = np.arange(0, XLEN, DIFF)
ptsy = np.arange(0, YLEN, DIFF)
xs, ys = np.meshgrid(ptsx, ptsy)
z_orig = np.random.random((XLEN + 1, YLEN + 1))
z_interped = np.zeros((round((XLEN) / DIFF), round(YLEN / DIFF)))

def lerp(a, b, t):
    return a + t * (b - a)

def lerp2(ld, rd, lu, ru, tx, ty): # Two-dimensional linear interpolation
    # left down, right down, left up, right up
    upmid = lerp(lu, ru, tx)
    dnmid = lerp(ld, rd, tx)
    return lerp(dnmid, upmid, ty)

for i in range(XLEN):
    for si in range(round(1 / DIFF)):  # step i
        for j in range(YLEN):
            for sj in range(round(1 / DIFF)):
                z_interped[i * round(1 / DIFF) + si][j * round(1 / DIFF) + sj] = lerp2(
                    z_orig[i][j], z_orig[i + 1][j], z_orig[i][j + 1],  z_orig[i + 1][j + 1], DIFF * si, DIFF * sj)


plt.imshow(z_interped, cmap=plt.cm.gray)
plt.savefig("./2d.png", dpi = 150, format = 'png')
plt.show()

It is easy to see that this noise looks unnatural; you can even vaguely make out the coordinate axes in the image. Although the entire image looks relatively random, close inspection shows that it is assembled from many small, square patches of color.

This happens because each key point has the same influence in every direction, while linear interpolation turns that influence into a diamond-like shape. The point in the center of the following image is a key point. Its random value is relatively low, so it is black, and we can see that the black region spreading outward from it has a diamond shape.

Changing this is also very simple: make a key point’s influence on its surroundings differ by direction. Since we need to represent a direction, vectors are a natural idea.

We now generate random unit vectors at each key point and denote them by $\vec{g_i}$ (the random vector generated at key point $i$ ), as follows

The question is now how to use these random vectors to produce different influences in different directions. One natural idea is to consider the position of a point relative to a key point. We can denote this displacement vector by $\vec{d_i}$ (the displacement from key point $i$ ), as shown below

If $\vec{d_i}$ and $\vec{g_i}$ point in similar directions, we can make the point brighter. Conversely, if $\vec{d_i}$ and $\vec{g_i}$ point in opposite directions, the point should be darker.

This effect can be achieved with a dot product, which is essentially the length obtained by projecting $\vec{d_i}$ onto $\vec{g_i}$ . The result is negative for opposite directions, positive for the same direction, and zero for perpendicular directions.

We record this dot product:

$\vec{v_i} = \vec{d_i} \cdot \vec{g_i}$

We can then linearly interpolate among the four surrounding points in the same way as value noise. In other words, we treat $\vec{v_i}$ as the value that used to reside at a key point in value noise. This value now changes for each position.

The following image demonstrates Perlin noise. Different arrows represent different $\vec{g_i}$ values; smaller values are bluer and larger values are yellower

Pay attention to the three boxes in the image.

Most of the red box is yellow because the displacement vectors of the points in this region have directions similar to the key point’s random vector.
Most of the yellow box is blue because the tail of the random vector at its lower-left key point points toward this region. In other words, the displacement vectors in this region are opposite to the random vector at that key point.
Most of the green box is yellow. Although this region lies in the direction opposite to the random vector of the key point on the left, linear interpolation is present and the region is closer to the random vector on the right, so it is influenced more strongly by the vector on the right.

It is very clear that the noise generated by Perlin noise does not have the blocky appearance of value noise.

Turbulence

Consider the following implementation of turbulence:

double turb(const point3& p, int depth=7) const {
    auto accum = 0.0;
    auto temp_p = p;
    auto weight = 1.0;

    for (int i = 0; i < depth; i++) {
        accum += weight*noise(temp_p);
        weight *= 0.5;
        temp_p *= 2;
    }

    return fabs(accum);
}

The turb function itself is relatively easy to understand: it superimposes Perlin noise at many frequencies using certain weights. The final fabs appears to keep the returned value within the $[0, 1]$ range, but it actually has another purpose. For example, if we replace the last line with return (accum + 1) * 0.5, the returned value also lies within the range, but the result looks very different from the original implementation.

In the following figure, the blue line is the graph of $y = \sin x$ , the red line is the graph of $y = |\sin x|$ , and the green line is $y = \left(\sin\left(x\right)+1\right)\times 0.5$ :

With the green-line correction, regions that were originally dark remain dark afterward, and vice versa. With the red-line correction, only regions that originally had medium brightness, or transitions between light and dark, become dark; both dark and bright regions become brighter after correction. Comparing a characteristic region of the two materials in the book makes the behavior of the red correction clearer:

The black border in the left image looks as if it outlines the black region in the right image, which agrees with the prediction that only the transition region becomes darker.

Some Questions

The maximum value returned by noise(p) in the code is 1, and the initial value of weight is also 1. Therefore, abs(accum) can be greater than 1. This clearly makes no sense, because a ray cannot become brighter after hitting an object (other than a light source). I previously emailed the author of this blog about the question, but he said that he did not know either; perhaps values greater than 1 are simply rare because of probability.

I then looked at Ken Perlin’s 1985 SIGGRAPH paper^[3]. It contains neither a very rigorous description nor actual code, although the basic idea is clear. One thing I found strange is that the entire paper never says that the new noise algorithm is intended to improve value noise. It mainly focuses on the fact that the effect of Perlin noise is unaffected by various spatial transformations (did he invent a noise algorithm independent of spatial transformations and improve value noise along the way? That would be rather absurd):

Noise()
In order to get the most out of the PSE and the solid texture approach we have provided some primitive stochastic functions with which to bootstrap visual complexity. We now introduce the most fundamental of these. Noise() is a scalar valued function which takes a three dimensional vector as its argument. It has the following properties :
Statistical invariance under rotation (no matter how we rotate its domain, it has the same statistical character)
A narrow bandpass limit in frequency (its has no visible features larger or smaller than within a certain narrow size range)

Appendix. Turbulence
A suitable procedure for the simulation of turbulence using the Noise() signal is :
1
2
3
4
5
6
7
function turbulence(p)
  t = 0
  scale = 1
  while (scale > pixelsize)
      t += abs(Noise(p / scale) * scale)
      scale /= 2
  return t

The turbulence pseudocode is basically no different from the version in the book. For the Noise() function, however, Perlin only says that it takes the position of a point and returns a scalar, without specifying the scalar’s range, so the issue remains rather puzzling.

Nevertheless, a sentence later in the paper still makes it seem that he intended to return a value in the $[0, 1]$ range (he mentions using a color such as $[1, 1, 1]$ ):

By evaluating Noise() at visible surface points of simulated objects we may create a simple “random” surface texture (figure Spotted.Donut) :
color = white * Noise(point)

This question troubled me for quite a long time. If you know the correct explanation, you are welcome to leave it in the comments. I also plan to ask about it on Stack Overflow after a while; if I get an answer, I will update this blog post.

Instance Transformations

Rotation Matrices

Formula Derivation

When I first saw the following formulas in the book, I was rather confused:

$x^\prime = \cos(\theta) - \sin(\theta) \cdot y \\y^\prime = \sin(\theta) + \cos(\theta) \cdot y$

After searching online, I found that this is actually a rotation matrix. The formula is derived as follows (the preceding formula describes rotation around the z-axis, which we can understand simply as a rotation matrix in a two-dimensional plane)

$x = r\cos\phi \\y = r\sin\phi$

Add $\theta$ to the original angle:

$x^\prime = r\cos(\phi + \theta) \\y^\prime = r\sin(\phi + \theta)$

Use the following two angle-sum formulas:

$\cos(\phi + \theta) = \cos\phi\cos\theta - \sin\phi\sin\theta \\\sin(\phi + \theta) = \sin\phi\cos\theta + \cos\phi\sin\theta$

Substitute the polar-coordinate form of $(x^\prime, y^\prime)$ :

$\begin{align*} x^\prime &= r(\cos\phi\cos\theta - \sin\phi\sin\theta) \\ x^\prime &= (r\cos\phi)\cos\theta - (r\sin\phi)\sin\theta \\ x^\prime &= x\cos\theta - y\sin\theta \end{align*}$

$\begin{align*} y^\prime &= r(\sin\phi\cos\theta + \cos\phi\sin\theta) \\ y^\prime &= r(\sin\phi)\cos\theta + r(\cos\phi)\sin\theta \\ y^\prime &= y\cos\theta + x\sin\theta \\ &= x\sin\theta + y\cos\theta\end{align*}$

Some Explanations

Rotation around the $x$ -axis is basically no different from this, but rotation around the $y$ -axis is rather puzzling. Rotations around the other two axes both have the form $\cos - \sin$ , $\sin + \cos$ , but for rotation around the $y$ -axis alone, the form becomes $\cos + \sin$ and $-\sin + \cos$ .

Because the sign of $\sin$ changes in a rotation around $y$ , it is obvious that we are actually rotating not by $\theta$ but by $-\theta$ . This is because the rotation direction we want is “different” from the rotation direction in a right-handed coordinate system.

That statement is vague, so we can proceed one step at a time and first determine the direction in which we want to rotate:

           y+
           |
           | 
           |
x- ------- z --------- x+
           |
           |
           |
           y-

This diagram shows a right-handed coordinate system viewed along the $z$ -axis. Note that the positive direction of the $z$ -axis points toward the observer. Clearly, if I say that I want to rotate something by $90^\circ$ around the $z$ -axis, I expect it to move from the positive $x$ direction to the positive $y$ direction. Equivalently, $y+ \to x-$ , $x- \to y-$ , and $y- \to x+$ ; in short, the rotation is counterclockwise.

Now consider rotation around the $x$ -axis:

           y+
           |
           | 
           |
z+ ------- x --------- z-
           |
           |
           |
           y-

Again, the $x$ -axis points toward the observer, so the rotation is counterclockwise, from $y+$ to $z+$ .

Now include the formulas and see whether they agree with our expectation—that is, a rotation from $y+$ to $z+$ .

Suppose the current point is $(x, y, z) = (0, 1, 0)$ (that is, it lies on $y+$ ). After a rotation of $90^\circ$ , $(x^\prime, y^\prime, z^\prime)$ should lie on $z+$ , namely at $(0, 0, 1)$ .

First consider the formula for $y^\prime$ :

$\begin{align*} y^\prime &= \cos \theta \cdot y - \sin\theta \cdot z \\ &= \cos(90) \cdot 1 - \sin(90) \cdot 0 \\ &= 0 - 0 = 0\end{align*}$

Next, consider $z^\prime$ :

$\begin{align*} z^\prime &= \sin \theta \cdot x + \cos \theta \cdot z \\ &= \sin(90) \cdot 1 + \cos(90) \cdot z \\ &= 1 + 0 = 1\end{align*}$

It appears to be correct.

Now consider rotation around the $y$ -axis:

           z-
           |
           | 
           |
x- ------- y --------- x+
           |
           |
           |
           z+

We find that if we still rotate counterclockwise by $90^\circ$ and start on $x+$ , the point should move to $z-$ . If we continue using the formula for the other two axes, however, it moves to $z+$ instead, as follows:

$\begin{align*} z^\prime &= \sin \theta \cdot x + \cos \theta \cdot z \\ &= \cos(90) \cdot 0 - \sin(90) \cdot -1 \\ &= 0 - (1 \cdot -1) = 1\end{align*}$

Sure enough, changing the sign of $\sin$ in the formula solves the problem.

What is special about rotation around $y$ ? Consider an example. For rotations around the other two axes, if the direction of the rotation angle is counterclockwise and the rotation goes from the lower-numbered axis to the higher-numbered axis (such as $x \to y$ or $y \to z$ ), the directions of those two axes are the same ( $x+ \to y+$ and $x- \to y-$ ).

For rotation around the $y$ -axis, if we rotate counterclockwise from the lower-numbered axis to the higher-numbered axis, the directions of those two axes differ ( $x+ \to z-$ and $x- \to z+$ ).

After all, trigonometric functions were originally designed for the Cartesian plane (the xy-plane). When they are applied to a plane with different signs, some adjustment is inevitably required.

You may now wonder whether changing to a left-handed coordinate system would solve the problem. The answer is both yes and no. Rotation around the $y$ -axis would indeed no longer require changing the sign of $\sin$ , but rotation around the $z$ -axis would require it. Reversing the direction of the $z$ -axis is equivalent to viewing the previous xy-plane from the opposite side, so a counterclockwise rotation from $x$ to $y$ becomes $-x \to +y$ or $+x \to -y$ .

Some Small Implementation Issues

To be updated.

Volumetric Fog

To be updated.

Multithreading

To be updated.

1. ↩
1.https://zhuanlan.zhihu.com/p/201012251 ↩
2. ↩
2. ↩
2. ↩
2.https://www.cnblogs.com/leoin2012/p/7218033.html ↩
3.https://dl.acm.org/doi/pdf/10.1145/325165.325247 ↩
4.First, express $x$ and $y$ in polar coordinates: ↩
4.https://zhuanlan.zhihu.com/p/102814853 ↩

Ray Tracing in One Weekend Study Notes (2): Implementing the Camera Class

2022-09-06T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Implementing the Camera Class

In addition to the Lambertian material, another interesting part of RTOW is the implementation of the camera class, especially the depth-of-field portion.

Positioning the Camera

Let us first look at a relatively simple part of the camera class—the camera’s position. Three parameters are enough to determine the camera’s position: the position of the camera itself (lookfrom), the position at which the camera is taking a photograph (lookat), and a vector representing the position above the camera (vup). The figure in the book explains this very well:

In the constructor, we need to convert these three parameters into three parameters representing the camera’s orientation, and also handle the focal length, aperture, and FOV. The book does not spend much space on this part. It took me quite a while to understand it, so below are some thoughts about the implementation in the book.

Because I made some small changes to the code in the book (mainly naming?), I will first paste the code:

#pragma once
#include "rtow.h"

// f8 here is double (a float occupying eight bytes).
class camera {
   public:
    camera(vec3 lookfrom, vec3 lookat, vec3 vup = vec3(0, 1, 0), f8 vfov = 90,
           f8 asp_ratio = 16.0 / 9.0, f8 aperture = 0, f8 foc_len = 1) {
        f8 deg_fov = deg2rad(vfov);
        f8 half_hei = tan(deg_fov / 2);  // Opposite side divided by adjacent side, while the adjacent side is 1.
        f8 half_wid = half_hei * asp_ratio;

        cam_z = (lookfrom - lookat).unit_vec();
        // z points opposite to the direction in which the lens is pointing.
        cam_x = cross(vup, cam_z).unit_vec();  // Perpendicular to both vup and z.
        cam_y = vup.unit_vec();

        horizon = 2 * half_wid * cam_x * foc_len; // The horizontal and vertical frame vectors on the focal plane.
        vertic = 2 * half_hei * cam_y * foc_len;

        orig = lookfrom;

        lower_left_corner = orig - horizon / 2 - vertic / 2 - cam_z * foc_len; // The lower-left corner of the focal plane.

        len_radius = aperture / 2;
    }

    inline ray get_ray(f8 x, f8 y) const {
        // The ranges of x and y are [0, 1].
        // Coordinates of the pixel on the camera sensor.
        vec3 rd = len_radius * rand_unit_disk();
        vec3 offset = cam_x * rd.x() + cam_y * rd.y(); 

        ray r;
        r.orig = orig + offset;
        r.dir = lower_left_corner + x * horizon + y * vertic -
                orig - offset;  
            // Produce a vector from orig + offset to the corresponding pixel.
            // This is because a ray is represented by orig + t * dir.
        return r;
    }

    vec3 orig;               // Camera position.
    vec3 lower_left_corner;  // Lower-left corner of the image.
    vec3 horizon, vertic;    // Image dimensions (or the size of the plane at distance foc_len from the camera).
    vec3 cam_x, cam_y, cam_z;// Camera orientation.
    f8 len_radius;           // Aperture radius.
};

The following figure describes the relationships among the variables in the code:

It is relatively easy to understand the code using this figure.

The following code first calculates two variables, half_hei and half_wid:

1
2
3

f8 deg_fov = deg2rad(vfov);
f8 half_hei = tan(deg_fov / 2);  // Opposite side divided by adjacent side, while the adjacent side is 1.
f8 half_wid = half_hei * asp_ratio;

They represent the size of the image seen at a distance of one unit in front of the camera. Next, the three vectors cam_x, cam_y, and cam_z need to be calculated as follows:

cam_z = (lookfrom - lookat).unit_vec();
// z points opposite to the direction in which the lens is pointing.
cam_x = cross(vup, cam_z).unit_vec();  // Perpendicular to both vup and z.
cam_y = vup.unit_vec();

cam_z represents a direction from lookat to loofrom, which is opposite to the position at which the camera is actually taking the photograph.
The calculation of cam_x uses the cross product of vectors. In three-dimensional space, if $u = v \times w$ , then $u$ is perpendicular to both $v$ and $w$ . Of course, there are two vectors satisfying this condition, and the right-hand rule can be used to determine which one. From the preceding definition, we can conclude that cam_x is perpendicular to both cam_z and vup (that is, cam_z).
cam_y is the unit vector of vup.

Although I roughly understood the geometric meaning of the cross product of three-dimensional vectors, I did not previously completely understand how it was derived. I found the explanation in the following blog particularly clear; even I, a beginner, could understand it:

https://www.cnblogs.com/qilinzi/archive/2013/05/09/3068158.html

The calculations of horizon, vertic, and lower_left_corner are relatively simple, so I will not explain them here; they are all marked in the figure.

Implementing Depth of Field

Depth of Field in Reality

To understand how a computer simulates depth of field, we first need a basic understanding of the structure of a camera lens, as follows:

Without a lens, light originating from point A can propagate in many directions, and each direction reaches a different position on the imaging plane. In the end, the color of each point on the imaging plane is contributed by many different rays, naturally producing a blurry image.

After adding a lens and considering point A again, we can observe that the rays originating from point A in every direction eventually converge at one specific point on the imaging plane, namely A’. The resulting image is clear.

More generally, a lens satisfies the following two conditions:

Rays emitted in different directions from the same point must converge at the same point after passing through the lens.
Rays emitted from different points on the same plane converge at different points after passing through the lens.

There is one prerequisite: the point must be on the focal plane of the camera. If the distance between a point and the camera’s imaging plane is not the focal length, the following occurs:

If the imaging plane is the green one, A1 is on the correct focal plane. If the imaging plane is the red one, A2 is on the correct focal plane.

For convenience, let us analyze A1. On the red imaging plane, we find that rays originating in two directions (horizontally and diagonally) converge at different points. On the green imaging plane, they converge at only one point.

Although they converge at different points, the degree of difference varies. Imagine moving A1 further to the left. Then the position of A1’ on the red imaging plane must become higher. Conversely, if A1 is moved to the right, the position of A1’ on the red imaging plane also falls, eventually converging at the correct point. If it continues moving right, the position of A1’ on the red imaging plane continues to fall. Eventually, the distance on the imaging plane between the rays originating horizontally and diagonally from A1 increases.

Alternatively, if we increase the size of the lens, more rays originating from A1 at different angles can enter the lens and then reach the imaging plane. In this case, the position of A1’ on the red imaging plane becomes higher. We can imagine that the lens has been pulled upward, and the triangle formed by the rays has also been pulled upward (I was too lazy to draw the figure myself, so I used an image from the Internet to explain it this way).

From the preceding figure, we can see that, theoretically, there is only one distance at which the camera can produce a sharp image; slightly more or less is no longer sharp. In reality, however, the resolving ability of the human eye is not that strong. The range of distances that can produce a sharp image when the camera takes a picture (that the human eye considers sharp) is called the depth of field, as follows:

We can think about the influence of the lens size, or radius, mentioned above from the perspective of depth of field. In reality, the radius of the lens does not change. The usual method is to add an adjustable “gate” to the lens—the aperture—to control the light entering the lens, as follows:

We can see that a larger aperture reduces the depth of field, and vice versa.

Actual Implementation

Earlier, we considered that rays originating from one point at different directions converge at different points on the imaging plane when the focal length is incorrect. During actual rendering, however, we consider the contributions of different rays to a particular pixel on the imaging plane.

Therefore, when the aperture is large, more rays from different directions should simultaneously contribute to one point on the imaging plane, producing a blur. See the figure below, which shows the implementation of depth of field in RTOW:

In the code, we randomly select a point on the aperture and then trace a ray from the aperture to the corresponding pixel on the focal plane. Finally, we average the contributions of the sampled rays. The larger the aperture, the smaller the depth of field. Moreover, because every ray must pass through the corresponding point on the focal plane, we can ensure that the focal plane is always sharp.

Compared with how a real lens works, this is quite different, but it achieves the same effect. This is also due to the characteristics of ray tracing: it traces “backward” starting from the pixel. Therefore, we do not focus on the issue of rays emitted by one point in a real lens converging at different positions on the imaging plane. Instead, we think from another angle: how many rays emitted by different points affect one pixel. I have to say that this implementation in the book is really impressive.

References:

https://jishuin.proginn.com/p/763bfbd2e03f

Ray Tracing in One Weekend Study Notes (1): Lambertian Materials and Radiometry

2022-08-31T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

~~Recently~~ (it has been a week since I finished RTOW, and I am only writing these notes now—truly lazy) I spent some time finishing Ray Tracing in One Weekend (hereafter RTOW) ~~I really am not good enough; I did not finish this thing in one weekend~~ and also wrote the code.

The book is excellent, and the final rendering was beyond my expectations (the cover image). However, I previously knew nothing about computer graphics, so many basic concepts were unfamiliar. The book sometimes passes over basic knowledge, mathematical derivations, and proofs, so I am writing down my own thought process here.

Implementing a Lambertian Material

The book creates a Lambertian diffuse material as follows:

class lambertian : public material {
   public:
    lambertian(const color& alb) : albedo(alb) {}
    virtual optional> 
    get_ray_out(const ray& r_in, const hit_rec& rec) const override {
        vec3 ref_dir = rec.norm + rand_unit_vec(); // Note this line.
        if(ref_dir.near_zero()) // If rand_unit_vec() equals -rec.norm.
            ref_dir = rec.norm;
        ray ref_ray(rec.hit_pt, ref_dir);
        return make_pair(ref_ray, albedo);    
    }
    color albedo;  // Albedo.
};

After a ray hits a diffuse material, the scattered ray starts at the hit point (rec.hit_pt) and its direction is a random unit vector plus the normal at the hit point.

Why add the normal? Why not simply choose a random vector in a hemisphere?

Radiometry

To answer this, we need some knowledge of radiometry. In ray tracing we care about the light received by a camera (or the human eye), so the explanation below uses the camera’s perspective.

Basic Units

We first need to consider exactly which physical quantity a camera sensor receives. Clearly, it receives energy—or, in other words, a number of photons arriving at the sensor. We therefore regard the physical quantity received by the sensor as radiant energy, denoted by $Q$ and measured in joules.

Energy alone, however, does not represent an object’s brightness very well. After all, photographing the same scene with an exposure of one minute and an exposure of $\frac{1}{100}$ second will certainly produce different results.

Although the sensor ultimately receives energy, we can continue obtaining more energy simply by exposing, or integrating, for a longer time while holding the camera.

Naturally, this suggests dividing the received energy by the time spent collecting it. This gives the unit known as radiant flux:

$\Phi = \frac{\mathrm{d}Q}{\mathrm{d}t}$

That is, radiant flux is the energy the sensor can receive per unit time.

Conversely, it can also describe the energy transmitted by a light source per unit time.

This still cannot completely describe an object’s brightness. If we use a larger sensor in the camera, the larger sensor can receive more energy per unit time.

Using a larger sensor while observing something cannot change the brightness of the object itself. We must therefore divide the received radiant flux by area, obtaining radiant flux per unit area. This unit is called irradiance.

For a light source, using a larger emitting surface can likewise provide more total radiant flux, while the flux provided per unit area remains unchanged.

$E = \frac{\rm{d}\Phi}{\rm{d}A}$

We find that as the viewing distance increases, a larger area is required to collect the same luminous flux, so irradiance becomes smaller. This plainly conflicts with common experience: as distance increases, the brightness we observe does not decrease significantly. When attenuation does occur, it is mainly because light encounters many tiny particles while propagating.

What is happening? Intuitively, although a more distant observer receives less radiant flux, the object also appears smaller to the human eye.

For example, consider a lamp with a very large area and another lamp with a very small area. If they emit the same radiant flux, the smaller lamp is clearly brighter.

Thus, the flux received directly by the eye decreases, but the observed area of the object decreases correspondingly. These two changes cancel, leaving the observed brightness unchanged. We therefore need a physical quantity that describes the apparent size of the object seen by the eye. Dividing irradiance by this quantity will then genuinely describe brightness. That quantity is the solid angle.

We can imagine the eye’s field of view as a sphere whose center is the eye. Every point on the sphere is consequently the same distance from the eye. If many equally sized objects are placed on the surface of this sphere, they are the same distance from the eye and therefore appear to have the same size.

Objects at different distances can all be projected onto this sphere. An object whose projection occupies a larger area on the sphere appears larger to the eye.

From the perspective of a light source, we sometimes want to focus on the source’s effect in a particular direction—how much it illuminates that direction and how much radiant flux it provides. A solid angle can also be introduced for that analysis.

The solid angle is therefore defined as the projected area of an object on a unit sphere, whose radius is 1.

It is calculated as follows and measured in steradians (sr):

$\Omega = \frac{a}{R^2}$

Here, $a$ is the area projected onto a sphere, which need not be a unit sphere, and $R$ is the sphere’s radius.

With the solid angle, we can genuinely describe the apparent size of the object seen by the eye. Modifying irradiance further gives the physical quantity called radiance:

$L_\theta = \frac{\rm{d} \Phi}{\rm{d}A\cos(\theta)\rm{d}\Omega}$

In this formula, $A$ is the area of the photosensitive surface element, and $\Omega$ is the solid angle. The factor $\cos(\theta)$ calculates the area of an object parallel to the spherical surface, as shown below:

Here, $\theta$ is the angle between the object’s surface normal and the sphere’s normal. When $\theta$ is $0$ , $\rm{d}A\cos(\theta)$ is largest. When $\theta$ is $\frac{\pi}{2}$ , the object’s surface is perpendicular to the spherical surface, so rays emitted from the sphere do not intersect the object at all, and $\rm{d}A\cos(\theta)$ is 0.

Radiance already describes the brightness characteristics of most objects quite perfectly. The preceding discussion, however, concerns area light sources or sensors with a nonzero area. A point light has no area, so radiance is meaningless for it because the formula divides by area.

At other times, we may not care about the areas of the light source and sensor and may simply want to know the radiant flux emitted or received in a direction. We then need another physical quantity—radiant intensity—which is obtained by removing the division by area from radiance:

$I = \frac{\rm{d}\Phi}{\rm{d}\Omega}$

Lambert’s Cosine Law

Mathematically:

$I_\theta=I_n\times\cos\theta,$

where $I_n$ is the intensity when the surface normal is parallel to the ray. The larger the angle between the observer’s normal and the ray, the less flux is received. The cosine computes the area of the surface projected onto a plane perpendicular to the ray.

Lambertian Materials and Diffuse Reflection

Wikipedia describes a Lambertian radiator as a source whose spatial intensity distribution follows the cosine law; its intensity decreases with angle. A Lambertian surface, however, has the same radiance from every viewing direction.

The definitions explain why these statements are consistent:

$L_\theta=\frac{\mathrm d\Phi}{\mathrm dA\cos\theta\mathrm d\Omega},\qquad I_\theta=\frac{\mathrm d\Phi}{\mathrm d\Omega}\cos\theta.$

Thus the cosine cancels between numerator and denominator for radiance. Intensity depends on angle, but radiance does not. From the observer’s viewpoint, an oblique surface also appears smaller, which cancels the reduced flux.

Why Is the Code Written This Way?

Ray tracing follows rays backwards, from the camera toward the light. A camera point $B$ may receive contributions from many rays leaving an object point $A$ , so after tracing from $A$ to $B$ we must choose the next direction.

I understand this as tracing radiant intensity: each pixel has the same area, so its color depends on flux from a direction. Lambert’s cosine law must therefore be considered when measuring the contribution to $B$ .

The book samples each pixel repeatedly:

……
color pixel_color(0, 0, 0);
for (int s = 0; s < samples_per_pixel; ++s) {
    auto u = (i + random_double()) / (image_width-1);
    auto v = (j + random_double()) / (image_height-1);
    ray r = cam.get_ray(u, v);
    pixel_color += ray_color(r, world);
}
write_color(std::cout, pixel_color, samples_per_pixel);
……

There are two ways to model the cosine attenuation. Choose a random direction on the unit hemisphere and multiply by the cosine of its angle with the normal:

Or use $\cos\theta$ as the probability density function for choosing directions, eliminating the explicit attenuation:

The book chooses the second method. If $\cos\theta$ is treated as the length of a segment making angle $\theta$ with the normal and one endpoint is fixed at $B$ , the result is a circle tangent to $B$ (a sphere in three dimensions):

I do not currently know how to prove this, but it is correct. RTOW uses it by adding the hit-point normal to a random unit vector:

1	vec3 ref_dir = rec.norm + rand_unit_vec();

One remaining question is why the light received at $B$ is uniformly scattered around it. We are tracing radiant intensity, so should we calculate the angle between the surface normal and the camera and apply cosine attenuation?

As the angle increases, the surface area corresponding to one pixel also increases, cancelling the cosine attenuation. Since each sample chooses an arbitrary position inside the pixel, repeated samples cover the surface corresponding to that pixel. If we explicitly applied cosine attenuation, we would likewise take more samples from regions with larger angles, because those regions cover a larger area.

References:

1.http://www2.bren.ucsb.edu/~dturney/WebResources_13/RemoteSensing/TheLightHandbook.pdf ↩

[MIT 6.s081] Xv6 Lab 11: Mmap Record

2022-08-21T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Update on 2022/9/14: I recently put the lab code on GitHub. If you need a reference, you can find it here:

https://github.com/ttzytt/xv6-riscv

The different branches contain the different labs.

This is the final lab. I have finally finished them all!

Lab 11: mmap

Description

This lab implements a subset of the mmap() and munmap() system calls commonly found in UNIX operating systems. These system calls map a file into user-space memory, allowing the user to modify and access the file directly through memory, which is much more convenient.

The definition of mmap() is:

1 2	void mmap(void addr, size_t length, int prot, int flags, int fd, off_t offset);

It maps the first length bytes of the file whose descriptor is fd into memory beginning at addr, with an additional offset, meaning that the mapping does not necessarily start at the beginning of the file.

If addr is zero, the system automatically allocates an unused memory region for the mapping and returns that address.

In this lab, we only need to support cases in which both addr and offset are zero, so we do not need to consider a user-specified memory address or file offset at all.

Both prot and flags are sets of flags. More specifically, prot has the following options:

PROT_NONE
PROT_READ
PROT_WRITE
PROT_EXEC

They specify which operations may be performed on the mapped file.

The flags argument determines whether changes made through the memory mapping must be written back to the file when the mapping is removed.

Its two options are MAP_SHARED and MAP_PRIVATE.

The definition of munmap() is:

1	int munmap(void *addr, size_t length);

It removes a file mapping of length bytes beginning at addr. One important restriction is that the function cannot “punch a hole” in the middle of a mapped range. It may remove only a portion at the beginning or end, or the complete mapping.

That may not be entirely clear. Suppose the mapped range is $[1,100]$ . If we want to unmap the range $[l,r]$ , it must satisfy either $l=1 \And r\le100$ or $l\ge1 \And r=100$ .

Overall approach

First, we must decide where in a user process to place memory-mapped files. The memory layout of a user process is shown below:

Initially, I wanted to allocate memory for mappings in the same way as sbrk():

uint64
sys_sbrk(void)
{
  int addr;
  int n;

  if(argint(0, &n) < 0)
    return -1;
  addr = myproc()->sz;
  if(growproc(n) < 0)
    return -1;
  return addr;
}

In other words, allocate additional heap space to the process and put the file there. Although this would be easy to implement, closer consideration reveals many problems. We assume that all memory below myproc()->sz is freely available to the user, and that is precisely the memory allocated by malloc().

If a mapped file were placed there, the same space could easily be allocated by malloc() and then overwritten.

Furthermore, after the file is unmapped, the PTE for the mapped location is set to zero. If the user later accesses the corresponding memory, another page fault occurs and must be handled, which is clearly rather complicated.

We can avoid conflicts with the process heap by allocating memory for file mappings in the opposite direction. That is, begin at the trapframe and allocate mapping regions downward.

Following the lab hints, we can add a VMA (virtual memory area) structure to the kernel’s process structure. A VMA stores metadata about one file mapping, such as its starting address, length, and mapped file. This metadata makes the mappings much easier to manage.

To support a certain number of simultaneous file mappings, struct proc must contain that many VMAs. The hint recommends sixteen.

File mappings must also use lazy allocation; otherwise, copying a large file all at once would be very expensive. The file is copied into memory only after the user process triggers a page fault.

Finally, mapped files must remain available after fork(). This part is relatively simple: we only need to copy the VMA. The child page table does not contain the corresponding mapping, so accessing an address recorded in the VMA triggers a page fault. At that point, the required portion of the file can be copied into memory.

Code

Note: this lab does not register the system calls or mmaptest for us. Follow the same procedure as in Lab 2. I will not repeat it here; if necessary, see this article.

struct mmap_vma:

// in proc.h
struct mmap_vma{
  int in_use;      // Whether this VMA structure represents an active file mapping
  uint64 sta_addr; // Starting address
  uint64 sz;       // Mapping size
  int prot;
  struct file* file; // Mapped file
  int flags;         // map_shared or map_private
};

#define VMA_SZ 16

struct proc {
  ……
  struct mmap_vma mmap_vams[VMA_SZ];
}

sys_mmap():

This call does not allocate physical memory. It calls get_mmap_space() to find an unused entry in mmap_vams and an available virtual region for the file mapping, then initializes the VMA structure.

It must also increment the reference count of the mapped file. Without this increment, the file would be closed when its reference count reached zero, leaving us unable to copy its contents into memory during lazy allocation.

// in sysfile
uint64 
sys_mmap(){
  uint64 addr, length, offset; // Only zero is supported for addr and offset
  int prot, flags, fd;
  struct file* file;
  //void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);
  // This really has a lot of arguments...
  try(argaddr(0, &addr), return -1)
  try(argaddr(1, &length), return -1)
  try(argint(2, &prot), return -1)
  try(argint(3, &flags), return -1)
  try(argfd(4, &fd, &file), return -1) // Obtain both the file and its descriptor
  try(argaddr(5, &offset), return -1)
  // Read the arguments
  struct proc* p = myproc();
  if(addr || offset) // This mmap subset does not support a custom address or offset
    return -1;
  if(!file->writable && (prot & PROT_WRITE) && (flags & MAP_SHARED))
    return -1;
  // The file itself is not writable, but PROT_WRITE was requested

  int unuse_idx = -1;
  uint64 sta_addr = get_mmap_space(length, p->mmap_vams, &unuse_idx);

  if(unuse_idx == -1)
    return -1;
  if(sta_addr <= p->sz) // No memory remains for mmap
    return -1;
  struct mmap_vma* cur_vma = &p->mmap_vams[unuse_idx];
  cur_vma->file = file;
  cur_vma->in_use = 1;
  cur_vma->prot = prot;
  cur_vma->flags = flags;
  cur_vma->sta_addr = sta_addr; 
  cur_vma->sz = length;
  filedup(file); // Increment the reference count
  return cur_vma->sta_addr;
}

get_mmap_space():

This function must locate an available memory region for a new file mapping, so we need to choose an allocation strategy. The safest method is to find the lowest virtual address used by all existing VMAs and use that position as the end of the new mapped region. This can never create a collision, but it also has a drawback:

First, to simplify unmapping, we do not allow two file mappings to share one page frame; otherwise, kfree() would release both at once.

Second, always allocating below the lowest virtual address may cause the mapping region to continue growing downward even when a usable hole exists. In some uncommon situations this strategy could reduce the memory available to the user heap. In an extreme case it could cause a problem, although this is very unlikely because MAXVA is normally enormous and at least larger than physical memory.

In any case, I had time to spare and wrote code that handles this situation. It uses two nested loops, each traversing all VMAs. See the comments for the details.

// in sysfile.c
uint64
get_mmap_space(uint64 sz, struct mmap_vma* vmas, int* free_idx){
  *free_idx = -1;
  
  // Return an address at which a new file mapping can be stored (its starting address).
  // Prefer a gap between VMA slots; if no gap exists, map below all existing regions.
  // A quicksort could be used here, but I was lazy...
  uint64 lowest_addr = TRAPFRAME;
  
  struct mmap_vma tmp; // Upper boundary; as in the diagram, the top may contain no mapped region
  tmp.sta_addr = TRAPFRAME, tmp.sz = 0;

  for(int i = 0; i <= VMA_SZ; i++){
    // Assume PGROUNDDOWN(sta_addr) of vmas[i] is the end of the new file mapping
    if(vmas[i].in_use == 0 && i != VMA_SZ){
      *free_idx = i;
      continue;
    } 
    uint64 ed_pos = i != VMA_SZ ? PGROUNDDOWN(vmas[i].sta_addr) 
                                : tmp.sta_addr;

    lowest_addr = ed_pos < lowest_addr ? ed_pos : lowest_addr; // Take the minimum
    
    for(int j = 0; j < VMA_SZ; j++){
      // Assume the new mapping begins above sta_addr + sz of vmas[j], the end of vmas[j]
      if(vmas[j].in_use == 0 && i != VMA_SZ) continue;

      uint64 st_pos = i != VMA_SZ ? vmas[j].sta_addr + vmas[j].sz 
                                  : tmp.sta_addr + tmp.sz; // This position is necessarily page-aligned
                                  
      if (ed_pos <= st_pos) continue; 
      // Skip here rather than checking below because unsigned subtraction would be incorrect
      if (ed_pos - st_pos >= sz){
        // The interval [st_pos, ed_pos)
        return st_pos;
      }
    }
  } 

  return lowest_addr - sz;
}

All mappings created so far are lazily allocated, so we need a function that handles page faults.

mmap_fault_handler():

There is a slightly troublesome corner case here. If the mapping size requested by the user exceeds the size of the file itself, the remaining mapped region must be filled with zeroes; otherwise, mmaptest() will not pass.

Another point is that after a page fault we allocate and map only one page, rather than mapping the entire file at once.

// in trap.c
int 
mmap_fault_handler(uint64 addr){
  struct proc* p = myproc();
  struct mmap_vma* cur_vma;
  if((cur_vma = get_vma_by_addr(addr)) == 0){
    // Find which file mapping contains this address.
    // Zero means it belongs to none of them.
    return -1;
  }

  if(!cur_vma->file->readable && r_scause() == 13 && cur_vma->flags & MAP_SHARED){
    DEBUG("mmap_fault_handler: not readable\n");
    return -1;
  } // Read fault
    
  if(!cur_vma->file->writable && r_scause() == 15 && cur_vma->flags & MAP_SHARED){
    DEBUG("mmap_fault_handler: not writable\n");
    return -1;
  } // Write fault
    

  uint64 pg_sta = PGROUNDDOWN(addr);
  uint64 pa = kalloc();
  if(!pa){
    DEBUG("mmap_fault_handler: kalloc failed\n");
    return -1;
  }

  memset(pa, 0, PGSIZE);

  int perm = PTE_U | PTE_V;
  if(cur_vma->prot & PROT_READ) perm |= PTE_R;
  if(cur_vma->prot & PROT_WRITE) perm |= PTE_W;
  if(cur_vma->prot& PROT_EXEC) perm |= PTE_X;
  // Impossible combinations were already rejected by mmap

  uint64 off = PGROUNDDOWN(addr - cur_vma->sta_addr); 
  // off is the number of page frames to skip when copying the file

  ilock(cur_vma->file->ip);
  int rdret;
  if((rdret = readi(cur_vma->file->ip, 0, pa, off, PGSIZE)) == 0){
    iunlock(cur_vma->file->ip);
    return -1;
  }

  iunlock(cur_vma->file->ip); // Do not put it because this file will be used again later;
                              // it can be put during unmap
  mappages(p->pagetable, pg_sta, PGSIZE, pa, perm);
  return 0;
}

get_vma_by_addr():

This helper is used by the preceding fault handler and returns the VMA containing a given address:

struct mmap_vam* 
get_vma_by_addr(uint64 addr){
  struct proc* p = myproc();
  for(int i = 0; i < VMA_SZ; i++){
    if(p->mmap_vams[i].in_use && addr >= p->mmap_vams[i].sta_addr && addr < p->mmap_vams[i].sta_addr + p->mmap_vams[i].sz){
      // Determine whether this address lies inside the file-mapped region
      return p->mmap_vams + i;
    }
  }
  return 0;
}

usertrap():

// in trap.c
……
if(r_scause() == 8){
  // system call

  if(p->killed)
    exit(-1);

  // sepc points to the ecall instruction,
  // but we want to return to the next instruction.
  p->trapframe->epc += 4;

  // an interrupt will change sstatus &c registers,
  // so don't enable until done with those registers.
  intr_on();

  syscall();
} else if((which_dev = devintr()) != 0){
  // ok
} else if ((r_scause() == 13 || r_scause() == 15)){
  try(mmap_fault_handler(r_stval()), bad = 1)
}
else{
  bad = 1;
}

if (bad){
  printf("usertrap(): unexpected scause %p pid=%d\n", r_scause(), p->pid);
  printf("            sepc=%p stval=%p\n", r_sepc(), r_stval());
  p->killed = 1;
}
……

We can now attempt to implement munmap(). If the VMA has the MAP_SHARED flag, modifications made in memory must be copied back to the file while the mapping is removed.

Because this process is relatively complicated, I wrote a separate mmap_writeback() function for it. We use the PTE_D flag in a PTE to determine whether a page of the file mapping has been modified. A modified page needs to be copied back.

This flag is not already defined, so define it in riscv.h according to the RISC-V manual:

1	#define PTE_D (1L << 7)

If the unmap address and length are not multiples of PGSIZE, this function becomes particularly complicated:

The region being unmapped may not cross a page-frame boundary; all of the removed memory then lies within one frame. That frame cannot be released, but the changed memory still needs to be copied back to the file.
If the ending address lies in the middle of a page frame, we need another case distinction. If that frame is the final page of the mapped region, it must be written back and then released. If it is not the final page, it cannot be released.

Possibly because of this complexity, every munmap() and mmap() call in mmaptest.c uses addr and len values that are multiples of PGSIZE. The lab hints also say that supporting the features used by mmaptest.c is sufficient. The following version therefore does not support nonmultiples of PGSIZE. I also wrote a version that does, but it has not been tested at all because I was too lazy to write an enhanced mmaptest.c. Perhaps I will do so when I have time.

Normal version:

// in vm.c
int
mmap_writeback(pagetable_t pt, uint64 src_va, uint64 len, struct mmap_vma* vma){
// Write dirty page frames back to the file and remove their mappings.
// Write back len bytes beginning at src_va.
  uint64 a;
  pte_t *pte;
  for(a = PGROUNDDOWN(src_va); a < PGROUNDDOWN(src_va + len); a += PGSIZE){
    if((pte = walk(pt, a, 0)) == 0){ 
      panic("mmap_writeback: walk");
    }
    if(PTE_FLAGS(*pte) == PTE_V)
      panic("mmap_writeback: not leaf");
    if(!(*pte & PTE_V)) continue; // Lazy allocation

    if((*pte & PTE_D) && (vma->flags & MAP_SHARED)){ 
      // Write back
      begin_op();
      ilock(vma->file->ip);
      uint64 copied_len = a - src_va;
      writei(vma->file->ip, 1, a, copied_len, PGSIZE);
      iunlock(vma->file->ip);
      end_op();
    }
    kfree(PTE2PA(*pte));
    *pte = 0;
  }
  return 0;
}

Version supporting values that are not multiples of PGSIZE (untested):

//in vm.c
int
mmap_writeback_na(pagetable_t pt, uint64 src_va, uint64 len, struct mmap_vma* vma){
  uint64 a;
  pte_t *pte;
  a = PGROUNDDOWN(src_va);

  if(a == PGROUNDDOWN(src_va + len)){ 
    // The unmapped portion lies within a single page frame
    begin_op();
    ilock(vma->file->ip);
    writei(vma->file->ip, 1, src_va, 0, src_va - a);
    iunlock(vma->file->ip);
    end_op();
  }

  for(; a < PGROUNDDOWN(src_va + len); a += PGSIZE){ // This part handles only complete pages;
                                                     // an ending in the middle of a page is not handled here
    if((pte = walk(pt, a, 0)) == 0){ 
      panic("mmap_writeback: walk");
    }
    if(PTE_FLAGS(*pte) == PTE_V)
      panic("mmap_writeback: not leaf");
    if(!(*pte & PTE_V)) continue; // Lazy allocation
    if((*pte & PTE_D) && (vma->flags & MAP_SHARED)){ 
      // Write back
      begin_op();
      ilock(vma->file->ip);
      // On the first iteration, a may be smaller than src_va
      uint64 copied_len = a - src_va;
      if(a < src_va){ 
        // The first page frame is incomplete.
        // This case still requires kfree because the range crosses a page-frame boundary.
        writei(vma->file->ip, 1, src_va, 0, src_va - a); 
      } else {
        writei(vma->file->ip, 1, a, copied_len, PGSIZE);
      } 
      iunlock(vma->file->ip);
      end_op();
    }
    kfree(PTE2PA(*pte));
    *pte = 0;
  }
  
  uint64 copied_len = a - src_va;
  uint64 len_left = vma->sz - copied_len;

  if (len_left){
    // Handle an unmap range ending in the middle of a page frame
    begin_op();
    ilock(vma->file->ip);
    writei(vma->file, 1, a, copied_len, len_left);
    if(len_left + a == vma->sz + src_va){ // The page frame where it stops is exactly the final one
      pte_t *pte;
      if((pte = walk(pt, a, 0)) == 0){ 
        panic("mmap_writeback: walk");
      }
      kfree(PTE2PA(*pte));
    }
    iunlock(vma->file->ip);
    end_op();
  }

  return 0;
}

By comparison, munmap() itself is fairly simple. One detail remains important: if no part of the mapped region remains after unmapping, the corresponding file is no longer needed. We therefore call fileclose() to decrement its reference count and close it.

We also must not forget the restriction on removing a mapping: it can be removed only from its beginning or end, not by punching a hole in the middle, as described at the start of this article.

// in sysfile.c
uint64
munmap(uint64 addr, uint64 len){
  struct proc* p = myproc();
  struct mmap_vma* cur_vma = get_vma_by_addr(addr);
  if(!cur_vma)
    return -1;

  if(addr > cur_vma->sta_addr && addr + len < cur_vma->sta_addr + cur_vma->sz){
    // Attempt to punch a hole in the middle
    return -1;
  }

  mmap_writeback(p->pagetable, addr, len, cur_vma);
 
  if(addr == cur_vma->sta_addr){ 
    // Removing from the starting position
    cur_vma->sta_addr += len;
  } 
  cur_vma->sz -= len;
  
  if(cur_vma->sz <= 0){
    // The entire mapped region is gone
    fileclose(cur_vma->file);
    cur_vma->in_use = 0;
  }
  return 0;  
}

You may notice that this function is not written as a system call. That is because we will also need to invoke it from inside the kernel. The system-call wrapper is:

uint64
sys_munmap(){
  // int munmap(void *addr, size_t length);
  uint64 addr;
  uint64 len;
  try(argaddr(0, &addr),  return -1)
  try(argaddr(1, &len), return -1)
  return munmap(addr, len);
}

The kernel needs to call munmap() because some processes may exit without unmapping their files. We must forcibly clean up those mappings to prevent memory leaks, and this cleanup can be placed in exit().

Why place it in exit() rather than in freeproc(), which actually releases the process slot? Observe that a process is passed to freeproc() by wait() as follows:

// in proc.c wait():
……
  for(;;){
    // Scan through table looking for exited children.
    havekids = 0;
    for(np = proc; np < &proc[NPROC]; np++){
      if(np->parent == p){
        // make sure the child isn't still in exit() or swtch().
        acquire(&np->lock);

        havekids = 1;
        if(np->state == ZOMBIE){
          // Found one.
          pid = np->pid;
          if(addr != 0 && copyout(p->pagetable, addr, (char *)&np->xstate,
                                  sizeof(np->xstate)) < 0) {
            release(&np->lock);
            release(&wait_lock);
            return -1;
          }
          freeproc(np); // Notice that freeproc is called only when the parent waits.
          release(&np->lock);
          release(&wait_lock);
          return pid;
        }
        release(&np->lock);
      }
    }
    ……
  }
……

If the parent process never calls wait(), these mapped files remain indefinitely and are never written back. Of course, a parent process ought to call wait(). The main reason I used exit() is that the lab hint says to do so, but the behavior just described may be why the hint makes that recommendation.

// in proc.c exit():
void
exit(int status)
{
  struct proc *p = myproc();

  if(p == initproc)
    panic("init exiting");

  // Release and write back mmap data before closing files
  for(int i = 0; i < VMA_SZ; i++){
    if(p->mmap_vams[i].in_use){
      try(munmap(p->mmap_vams[i].sta_addr, p->mmap_vams[i].sz), panic("exit: munmap"));
    }
  }

  // Close all open files.
  for(int fd = 0; fd < NOFILE; fd++){
    if(p->ofile[fd]){
      struct file *f = p->ofile[fd];
      fileclose(f);
      p->ofile[fd] = 0;
    }
  }
……
}

The final step of the lab is allowing a child process to access mapped files after fork(). As mentioned earlier, we only need to copy the VMA. Its sta_addr is a virtual address. When the child attempts to access it, a page fault occurs because that virtual address has not been mapped to a physical address.

In mmap_fault_handler(), we then find that the faulting address belongs to a file-mapped region. The handler allocates a physical page for that virtual page and copies the corresponding file data into it.

Of course, after fork() another process is using the mapped file, so filedup() must increment its reference count.

fork():

// in proc.c
……
  for (int i = 0; i < VMA_SZ; i++){
    if(p->mmap_vams[i].in_use){
      np->mmap_vams[i] = p->mmap_vams[i]; 
      filedup(p->mmap_vams[i].file);
      // Copy the VMA
    }
  }
……

Initially I had a small question here. The earlier call to uvmcopy() had already copied the memory, so would it not copy the VMAs too? If we copied them afterward, would that duplicate the mappings?

Reading the implementation resolved the question: uvmcopy() copies only memory below myproc()->sz:

// in vm.c
  for(i = 0; i < sz; i += PGSIZE){ // Notice the range here
    if((pte = walk(old, i, 0)) == 0)
      panic("uvmcopy: pte should exist");
    if((*pte & PTE_V) == 0)
      panic("uvmcopy: page not present");
    pa = PTE2PA(*pte);
    flags = PTE_FLAGS(*pte);
    if((mem = kalloc()) == 0)
      goto err;
    memmove(mem, (char*)pa, PGSIZE);
    if(mappages(new, i, PGSIZE, (uint64)mem, flags) != 0){
      kfree(mem);
      goto err;
    }
  }

After completing these changes, the lab passes. I wish everyone currently working on it an early AC as well:

Complaints

I absolutely have to complain about a bug here, although I do not even know where the bug belongs: xv6, QEMU, or the Makefile.

While debugging with GDB, I wanted to use macros, mainly PGROUNDDOWN() and PGROUNDUP(). I therefore added the -g3 compilation option to the Makefile as follows:

1	CFLAGS = -Wall -O -g3 -fno-omit-frame-pointer -ggdb -UFDEBUG

This caused one of the tests in usertest.c to fail with an immediate panic:

1
2
3

$ usertests writebig
usertests starting
test writebig: panic: balloc: out of blocks

Removing -g3 somehow made everything work normally. I could never have imagined that a compiler option might affect the number of blocks on the virtual disk. I spent an entire day debugging this because who would expect a compiler option to have such an effect? Eventually I used Git to compare the files on this branch with other branches and tested the differences one by one until I found it.

I reported the problem on the xv6-riscv GitHub repository. While browsing the issue tracker, I found something even more absurd:

https://github.com/mit-pdos/xv6-riscv/issues/59

Adding -O3 to the compiler options can apparently cause the same problem. I simply do not understand it.

[MIT 6.s081] Xv6 Lab 10: File System Record

2022-08-18T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Update on 2022/9/14: I recently put the lab code on GitHub. If you need a reference, you can find it here:

https://github.com/ttzytt/xv6-riscv

The different branches contain the different labs.

Lab 10: File System

Large files

Description

In xv6’s underlying implementation, a file is described by struct dinode, as follows:

struct dinode {
  short type;           // File type
  short major;          // Major device number (T_DEVICE only)
  short minor;          // Minor device number (T_DEVICE only)
  short nlink;          // Number of links to inode in file system
  uint size;            // Size of file (bytes)
  uint addrs[NDIRECT + 1];   // Data block addresses
};

Here we mainly care about the addrs field in this structure. It records the actual storage locations of the file. The first twelve entries of addrs point directly to blocks that store file data. The final entry is an indirect block: the block to which it points stores other pointers, and those pointers in turn point to the blocks that actually store the data. This may sound rather convoluted; it looks approximately like the following diagram:

We can calculate the maximum file size supported by xv6. A struct dinode occupies 64 B, and one disk block can store 1024 B of data.

The first twelve direct entries of addrs can therefore address $12 \times 1024B = 12288B$ of data.

The final indirect pointer points to a block filled with pointers to other disk blocks. That block can hold $1024B \div 4B = 256$ addresses.

Each address identifies an entire block, so this indirect entry of addrs supplies $256 \times 1024B = 262144B$ of storage.

Together they provide $262144B + 12288B = 274432B$ , which is $268KB$ .

This capacity is obviously very small. In this lab, we therefore need to add a doubly indirect block pointer to the inode.

A singly indirect block pointer is the last addrs entry just described: it points to a block, and the block pointers stored in that block point to the actual data blocks.

With a doubly indirect pointer, each pointer in the block addressed by addrs points to another block that stores pointers. This increases the available storage, somewhat like a multilevel page table, as illustrated below:

We can calculate the capacity supplied by this doubly indirect pointer. One block holds 256 block pointers, so the block addressed by addrs can contain the block numbers of 256 pointer blocks. The total is therefore $256\times256=65536$ blocks. At 1024 bytes per block this is 64 MB, a very substantial increase.

Approach

We need to modify the two functions bmap() and itrunc(). There is nothing particularly difficult to reason about, so I leave the detailed explanation in the code section.

Code

Because a doubly indirect index has been added, several macro definitions must first be changed:

#define NDIRECT 11 // Remove one direct index and add one doubly indirect index
#define NINDIRECT (BSIZE / sizeof(uint))
#define NBI_INDIRECT NINDIRECT * NINDIRECT // Blocks supplied by the doubly indirect index
#define MAXFILE (NDIRECT + NINDIRECT + NBI_INDIRECT) //

We must also modify struct dinode and struct inode. A dinode is stored on the disk itself, while an inode adds metadata to the dinode representation to make inode processing more convenient:

//in fs.h
// On-disk inode structure
struct dinode {
  short type;           // File type
  short major;          // Major device number (T_DEVICE only)
  short minor;          // Minor device number (T_DEVICE only)
  short nlink;          // Number of links to inode in file system
  uint size;            // Size of file (bytes)
  uint addrs[NDIRECT + 2];   // Data block addresses; changed to + 2 here
};

// in file.h
// in-memory copy of an inode
struct inode {
  uint dev;           // Device number
  uint inum;          // Inode number
  int ref;            // Reference count
  struct sleeplock lock; // protects everything below here
  int valid;          // inode has been read from disk?

  short type;         // copy of disk inode
  short major;
  short minor;
  short nlink;
  uint size;
  uint addrs[NDIRECT+2];// Changed to + 2 here
};

bmap():

This function accepts an inode pointer and bn, where bn means the index of a block within that inode, and returns the corresponding block number.

We need to add support for doubly indirect blocks to this function. To obtain a second-level indirect block, we can first obtain the corresponding first-level indirect block.

Much of the code can follow the existing handling of the singly indirect block.

// in fs.c
……
  bn -= NINDIRECT;
  // bn represents how many blocks remain

  if(bn < NBI_INDIRECT){
    if((addr = ip->addrs[NDIRECT + 1]) == 0) // If this block has not previously been allocated
      ip->addrs[NDIRECT + 1] = addr = balloc(ip->dev);    
    bp = bread(ip->dev, addr); // Short for buffer pointer
    a = (uint *)bp->data;

    uint idx_b1 = bn / NINDIRECT; // Obtain the index in addr of the first-level block corresponding to bn
    if((addr = a[idx_b1]) == 0){  // One first-level block covers 256 second-level blocks; check whether it exists
      a[idx_b1] = addr = balloc(ip->dev);
      log_write(bp); 
      // Mark this block as modified; it will later be updated in the disk log area.
      // It changed because a new block pointer was added to this pointer block.
    } 

    brelse(bp); // Release the cached block
    
    bp2 = bread(ip->dev, addr); // bp2 is the cache for the second-level block
    a = (uint *)bp2->data;
    uint idx_b2 = bn % NINDIRECT;
    if((addr = a[idx_b2]) == 0){
      a[idx_b2] = addr = balloc(ip->dev);
      log_write(bp2);
    }
    brelse(bp2);
    return addr;
  }
……

itrunc():

This function clears every block belonging to an inode; it can also be understood as deleting a file. Internally, it repeatedly calls brelse() and bfree().

Here, brelse() releases a cached block, while bfree() releases a disk block by modifying the data in the disk’s bitmap block.

As with bmap(), much of the implementation can follow the singly indirect index. The main idea resembles recursion: traverse every first-level block, check whether it contains data, and, if it does, traverse the second-level blocks referenced from it.

// in fs.c
……
  if(ip->addrs[NDIRECT + 1]){ // Determine whether the inode uses the doubly indirect index
    bp = bread(ip->dev, ip->addrs[NDIRECT + 1]);
    a = (uint*)bp->data;
    for (i = 0; i < NINDIRECT; i++){ // Traverse the first-level blocks
      if(a[i]){ // If data exists, traverse the second-level blocks within this first-level block
        struct buf* bp2 = bread(ip->dev, a[i]); // Obtain the corresponding cache for this block
        uint *a2 = bp2->data;
        for(j = 0; j < NINDIRECT; j++){
          if(a2[j])
            bfree(ip->dev, a2[j]); // a2[j] stores a block number; release that disk block here
        } 
 
        brelse(bp2); // Release the cached block
        bfree(ip->dev, a[i]); // Release the disk block
        // bp2 corresponds to a[i].
        // a[i] is the block number, while bp2 is the actual cached block.
      }      
    }
    brelse(bp); // Release the cache
    bfree(ip->dev, ip->addrs[NDIRECT + 1]); // Release the disk block
    ip->addrs[NDIRECT + 1] = 0;
  }
……

Symbolic links

Lab description

This exercise asks us to implement symbolic links, also called soft links. To be honest, I am still not completely clear about the essential difference between soft and hard links. A symbolic link is somewhat like a shortcut in Windows.

The implementation is actually simple. However, the hints supplied by this lab were not sufficient for me, so I was rather confused while doing it and eventually finished only after reading someone else’s blog.

First, a symbolic link is like a “pointer” to a file. When we open a symbolic link, the file to which it points is what actually gets opened. This lets a path in one directory open a file that is physically stored in a different directory.

Approach

How should this symbolic link be implemented? A symbolic link is itself a file. We only need to store the path of its target file in that file—or, more precisely, in its inode.

To achieve link following, open() must use the stored path to recursively find the final target file, because one symbolic link may point to another symbolic link.

But what if we want to open the symbolic link itself? That requires a new open() flag. Such flags specify settings for opening a file descriptor. We can add an O_NOFOLLOW flag meaning that the path stored in a symbolic link should not be followed recursively and that the link itself should be opened.

//in fcntl.h
#define O_RDONLY  0x000
#define O_WRONLY  0x001
#define O_RDWR    0x002
#define O_CREATE  0x200
#define O_TRUNC   0x400
#define O_NOFOLLOW 0x800

An inode is an abstraction over the various kinds of data stored on disk. To determine what an inode actually contains, we also need to define a new inode type:

//in stat.h
#define T_DIR     1   // Directory
#define T_FILE    2   // File
#define T_DEVICE  3   // Device
#define T_SYMLINK 4   // Symbolic link

One annoying detail in this exercise is that the sys_symlink() system call has not already been registered. As in Lab 2, it must be added to the various relevant files. I assume readers of this article have completed Lab 2, so I will not repeat that process. If you have not, see this article.

Code

sys_symlink():

As described above, a symbolic link is essentially a kind of file, but that file is itself represented by an inode. While writing the code, remember that all these operations are performed on an inode. In addition, all file-related system calls must be enclosed by begin_op() and end_op(). This means that every operation in that interval is first recorded in the logging system. For background, refer to the xv6 book and lectures.

uint64 sys_symlink(){
  char tar_path[MAXPATH], path[MAXPATH];
  try(argstr(0, tar_path, MAXPATH), return -1);
  try(argstr(1, path, MAXPATH), return -1);
  struct inode* ip;


  begin_op();
  ip = create(path, T_SYMLINK, 0, 0); // Create a file and return its inode (there are no comments, so I am not
                                      // entirely sure how this function is used; I inferred it from the implementation)
  if(ip == 0){
    end_op();
    return -1;
  }
  try(writei(ip, 0, tar_path, 0, strlen(tar_path)), end_op(); return -1); 
  // writei writes data to an inode; here it stores the path targeted by the symbolic link
  iunlockput(ip);
  // Standard operations after finishing with an inode:
  // first release the lock and then release this inode.
  // iput() for an inode is similar to brelse() for a cached block.
  // Both first decrement the reference count and then determine whether the object can truly be freed.
  end_op();
  return 0;
}

sys_open():

The following code at the beginning of sys_open() opens or creates the inode corresponding to the path supplied by the user and stores it in ip. The later code in sys_open() processes this ip to finish the open operation, but we do not need to consider that part yet.

\\ in sysfile.c
  if(omode & O_CREATE){
    ip = create(path, T_FILE, 0, 0);
    if(ip == 0){
      end_op();
      return -1;
    }
  } else {
    if((ip = namei(path)) == 0){
      end_op();
      return -1;
    }
    ilock(ip);
    if(ip->type == T_DIR && omode != O_RDONLY){
      iunlockput(ip);
      end_op();
      return -1;
    }
  }

For a symbolic link, the ip corresponding to the path supplied by the user is not the inode the user ultimately wants to open. We therefore need to follow the files referenced by symbolic links recursively and update ip. Note that the final ip must remain locked.

The code is as follows and is added after the preceding block:

\\ in sysfile.c
  if(!(omode & O_NOFOLLOW)){
    int rec_left = 10; // Recursion limit because symbolic links may form a cycle
    struct inode* next_file;
    while(rec_left && ip->type == T_SYMLINK){
      
      if(readi(ip, 0, path, 0, MAXPATH) == 0){
        iunlockput(ip);
        end_op();
        return -1;
      }

      if((next_file = namei(path)) == 0){
        // namei obtains an inode from a path
        iunlockput(ip);
        end_op();
        return -1;
      }
      iunlockput(ip); // We have finished using the file that stores the link
      ip = next_file;
      rec_left--;  
      ilock(ip); // Lock here rather than below the while loop because, even if this inode is not a symbolic link,
                 // we still need to hold the lock since the later processing code modifies the inode
    }
    if(rec_left <= 0){
      iunlockput(ip);
      end_op();
      return -1;
    }
  }

There is one particularly important detail here. While following symbolic links recursively, we need to stop when we reach a file that is not a symbolic link. This requires access to the inode’s type field. The check of this field must occur after ilock(ip). It took me a long time to discover this bug.

First, examine the code for ilock():

// Lock the given inode.
// Reads the inode from disk if necessary.
void
ilock(struct inode *ip)
{
  struct buf *bp;
  struct dinode *dip;

  if(ip == 0 || ip->ref < 1)
    panic("ilock");

  acquiresleep(&ip->lock);

  if(ip->valid == 0){
    bp = bread(ip->dev, IBLOCK(ip->inum, sb));
    dip = (struct dinode*)bp->data + ip->inum%IPB;
    ip->type = dip->type;
    ip->major = dip->major;
    ip->minor = dip->minor;
    ip->nlink = dip->nlink;
    ip->size = dip->size;
    memmove(ip->addrs, dip->addrs, sizeof(ip->addrs));
    brelse(bp);
    ip->valid = 1;
    if(ip->type == 0)
      panic("ilock: no type");
  }
}

The function first checks ip->valid. This valid field indicates whether the current inode’s data has been loaded from disk. If it has not, the function reads the disk first and loads the data into this inode.

In other words, accessing an inode before calling ilock() means the inode may still be empty, so the values read from it naturally have no meaning. This also reminds us once again that shared data accessed between threads must be locked.

After finishing these changes, the lab can finally pass. I also wish everyone working on this lab an early AC:

One reminder: if your tests run successfully in QEMU but make grade still fails, the likely cause is a timeout—perhaps my computer is simply too slow. In that case, increase the time limit in the Python grading program grade-lab-fs.

Summary

Array out-of-bounds errors and memory leaks are truly terrifying. The actual defect may have no apparent relationship whatsoever to the error reported by the system, making it nearly impossible to debug.

I will briefly describe some exceptionally foolish mistakes I made while doing this lab. The worst part is that debugging them consumed two entire afternoons.

At first, symlinktest caused a panic whose message was virtio_disk_intr status. I certainly did not know how to deal with something involving a virtual disk, so I stepped through the program and located the exact operation in symlinktest where the problem occurred:

r = symlink("/testsymlink/4", "/testsymlink/3");
if(r) fail("Failed to link 3->4");

close(fd1);
close(fd2); // The problem occurs here

fd1 = open("/testsymlink/4", O_CREATE | O_RDWR);
if(fd1<0) fail("Failed to create 4\n");

Here, symlinktest panicked immediately after calling close(fd2).

I stepped through it again and found that the call sequence when the failure occurred was approximately:

1	sys_close() -> fileclose() -> iput() -> itrunc() -> bread()：

I assumed that I had written itrunc() incorrectly. I even created a new branch and copied someone else’s itrunc(), but the problem remained.

Then I wondered whether it was some inexplicable issue, so I simply commented out that panic(). A new panic appeared, this time reporting freeing free block:

static void
bfree(int dev, uint b)
{
  struct buf *bp;
  int bi, m;

  bp = bread(dev, BBLOCK(b, sb));
  bi = b % BPB;
  m = 1 << (bi % 8);
  if((bp->data[bi/8] & m) == 0)
    panic("freeing free block"); // Here
  bp->data[bi/8] &= ~m;
  log_write(bp);
  brelse(bp);
}

Later, I also found that itrunc() had not released the singly indirect index blocks at all and instead attempted to release the doubly indirect index immediately because addrs[12] was nonzero. That made no sense: the singly indirect capacity should be exhausted before the doubly indirect index is used. Combined with the freeing free block panic, this made me fairly certain that some kind of out-of-bounds access was responsible.

Eventually I discovered that the problem was actually in struct inode:

struct inode {
  uint dev;           // Device number
  uint inum;          // Inode number
  int ref;            // Reference count
  struct sleeplock lock; // protects everything below here
  int valid;          // inode has been read from disk?

  short type;         // copy of disk inode
  short major;
  short minor;
  short nlink;
  uint size;
  uint addrs[NDIRECT+2];
};

I had changed addrs[NDIRECT + 1] to addrs[NDIRECT + 2] in dinode, but had forgotten to make the same change in inode.

Consequently, when I accessed addrs[12], I was actually accessing the dev field of the next inode. Things then became absurd: how could an inode’s doubly indirect index block possibly be block number one, the superblock?

I am actually curious why itrunc() did not free the superblock and exactly how this caused the virtual-disk panic. I am too tired to debug it further, but anyone interested can investigate.

That is enough. This completely broke me.

[MIT 6.s081] Xv6 Lab 9: Locks Record

2022-08-12T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Update on 2022/8/18: the second exercise in this article is not completely correct, and many other approaches exist. See my discussion with the author of this blog and the author’s new code based on that discussion.

If I have time later, I will revise the second part and add comments.

Update on 2022/9/14: I recently put the lab code on GitHub. If you need a reference, you can find it here:

https://github.com/ttzytt/xv6-riscv

The different branches contain the different labs.

Lab 9: locks

Memory allocator

Lab description

The lab description is again very long, so I will not reproduce a screenshot. Here is the general problem.

The original kalloc() uses one large lock and maintains a single freelist. Every program that allocates or frees memory must compete for that lock before modifying the list. The implementations of freelist, kfree(), and kalloc() are:

struct run {
  struct run *next;
};

struct {
  struct spinlock lock;
  struct run *freelist;
} kmem;

……

// Free the page of physical memory pointed at by v,
// which normally should have been returned by a
// call to kalloc().  (The exception is when
// initializing the allocator; see kinit above.)
void
kfree(void *pa)
{
  struct run *r;

  if(((uint64)pa % PGSIZE) != 0 || (char*)pa < end || (uint64)pa >= PHYSTOP)
    panic("kfree");

  // Fill with junk to catch dangling refs.
  memset(pa, 1, PGSIZE);

  r = (struct run*)pa;

  acquire(&kmem.lock);
  r->next = kmem.freelist;
  kmem.freelist = r;
  release(&kmem.lock);
}

// Allocate one 4096-byte page of physical memory.
// Returns a pointer that the kernel can use.
// Returns 0 if the memory cannot be allocated.
void *
kalloc(void)
{
  struct run *r;

  acquire(&kmem.lock);
  r = kmem.freelist;
  if(r)
    kmem.freelist = r->next;
  release(&kmem.lock);

  if(r)
    memset((char*)r, 5, PGSIZE); // fill with junk
  return (void*)r;
}

Multiple cores cannot call kalloc() or kfree() concurrently, greatly reducing memory-allocation performance.

Testing confirms that this global lock is a major bottleneck. Among all locks, kmem has the most waits and the most severe contention:

$ kalloctest
start test1
test1 results:
--- lock kmem/bcache stats
lock: kmem: #fetch-and-add 83375 #acquire() 433015
lock: bcache: #fetch-and-add 0 #acquire() 1260
--- top 5 contended locks:
lock: kmem: #fetch-and-add 83375 #acquire() 433015
lock: proc: #fetch-and-add 23737 #acquire() 130718
lock: virtio_disk: #fetch-and-add 11159 #acquire() 114
lock: proc: #fetch-and-add 5937 #acquire() 130786
lock: proc: #fetch-and-add 4080 #acquire() 130786
tot= 83375
test1 FAIL

The lab asks us to solve this problem. Its hint suggests assigning one freelist to each processor core. A core can then allocate a page without waiting on the globally contended lock. It still uses a lock, but contention is dramatically reduced.

Approach

This creates another problem. Some cores may have many free page frames while another has none. Even if the machine has enough free frames overall, the empty core cannot allocate one locally.

When a core has no available frames, it therefore needs to “steal” some from other cores.

The approximate pseudocode is:

struct {
  struct spinlock lock;
  struct run *freelist;
} kmems[NCPU];

void kalloc(){
    struct run* r = 0;
    push_off();
    int cpu = cpuid();
    pop_off();

    acquire(&kmems[cpu].lock);

    int stealed = 0;
    if(!kmems[cpu].freelist){
        for (i : kmems){
            acquire(&i.lock);
            while (i still has page frames && stealed < STEAL_CNT) {
                remove a page frame from i's freelist;
                add the removed frame to kmems[cpu].freelist;
            }
            if(stealed >= STEAL_CNT){
                break;
            }
            releae(&i.lock);
        }

    }
    r = kmems[cpu].freelist;

    if (r) {    
        kmems[cpu].freelist = r->next;
    }

    release(&kmems[cpu].lock);
    return r;
}

This looks reasonable and can even pass the tests, but it can deadlock, although the deadlock is extremely unlikely.

Notice the for (i : kmems) loop. While inside it, the core holds or attempts to acquire two locks: its own kmems[cpu].lock, and the i.lock of the core from which it wants to steal.

Suppose the processor has only two cores, a and b, and both have exhausted their free frames. Each first acquires its own lock and then attempts to steal from the other.

During stealing, each tries to acquire the other core’s lock. But a and b already hold their respective local locks, so both wait forever: a deadlock.

The same pattern can occur with more than two cores; two cores merely make the explanation easier.

To prevent it, a core must not simultaneously hold its own lock and another core’s lock.

That introduces a further issue. While one core is stealing pages and adding them to its local freelist, another may try to steal from it. Concurrent modification of the same list would corrupt it.

My solution works as follows.

When a core discovers that its list is empty, immediately release the local lock and begin stealing. Two locks would ordinarily be needed because several cores might modify a freelist at once. Instead, do not modify the local freelist while stealing. Remove available pages from other cores and record them in a candidate array. After reacquiring the local lock, scan that array and insert the stolen pages into the local list.

Because stolen pages are only recorded in the candidate array and are not yet in the local freelist, another core attempting to steal from this core still sees an empty list and does not modify it. No core changes the local list during stealing, so its lock is unnecessary during that interval.

One more issue is interrupts. During stealing, the core may hold no locks, allowing xv6 to enable interrupts. It could leave to run another process, which might call kalloc() again and begin a duplicate steal operation.

This leads to the following implementation.

Code

kinit():

struct {
  struct spinlock lock, stlk;
  struct run *freelist;
  uint64 st_ret[STEAL_CNT]; // Candidate array
} kmems[NCPU];

const uint name_sz = sizeof("kmem cpu 0");
char kmem_lk_n[NCPU][sizeof("kmem cpu 0")];

void
kinit()
{ 
  for(int i = 0; i < NCPU; i++){
    snprintf(kmem_lk_n[i], name_sz, "kmem cpu %d", i);
    initlock(&kmems[i].lock, kmem_lk_n[i]);
  }
  freerange(end, (void*)PHYSTOP);
}

kfree():

void
kfree(void *pa)
{
  struct run *r;

  if(((uint64)pa % PGSIZE) != 0 || (char*)pa < end || (uint64)pa >= PHYSTOP)
    panic("kfree");
  push_off();
  uint cpu = cpuid();
  pop_off();
  // Fill with junk to catch dangling refs.
  memset(pa, 1, PGSIZE);
  r = (struct run*)pa;
  acquire(&kmems[cpu].lock);
  r->next = kmems[cpu].freelist;
  kmems[cpu].freelist = r;
  release(&kmems[cpu].lock);
}

Whichever core is running the current process receives the released page in its own freelist. This is a simple allocation policy; better ones may exist, but I was lazy.

steal():

This newly added function scans every core’s freelist and places available pages into the current core’s candidate array, st_ret[STEAL_CNT]:

int steal(uint cpu){ // Return the number of pages stolen
  uint st_left = STEAL_CNT;
  int idx = 0; 

  memset(kmems[cpu].st_ret, 0, sizeof(kmems[cpu].st_ret));
  for(int i = 0; i < NCPU; i++){
    if(i == cpu)  continue;
    acquire(&kmems[i].lock);

    while(kmems[i].freelist && st_left){ 
      kmems[cpu].st_ret[idx++] = kmems[i].freelist;  
      kmems[i].freelist = kmems[i].freelist->next;  
      st_left--;
    }

    release(&kmems[i].lock);
    if(st_left == 0) { // STEAL_CNT pages have been stolen in total
      break;
    }
  }
  return idx;
}

kalloc():

When no local frame remains, call steal() and then truly add the stolen frames to freelist. Interrupts remain disabled for the entire kalloc() because enabling them could let two processes on one core execute steal() and steal the same pages twice.

void *
kalloc(void)
{
  struct run *r = 0;
  
  push_off();
  uint cpu = cpuid();   
  acquire(&kmems[cpu].lock);
  r = kmems[cpu].freelist;
  // r is the page frame that will be returned
  if(r){ 
    kmems[cpu].freelist = r->next;
    release(&kmems[cpu].lock);
    } else {
    release(&kmems[cpu].lock);
    int ret = steal(cpu); // kfree cannot occur during steal because interrupts are disabled
    // ret is the number of pages stolen
    if(ret <= 0){
      pop_off();
      return 0;
    }
    acquire(&kmems[cpu].lock);
    for(int i = 0; i < ret; i++){
      if (!kmems[cpu].st_ret[i]) break;
      ((struct run*)kmems[cpu].st_ret[i])->next = kmems[cpu].freelist; // Add the stolen page to the front of freelist
      kmems[cpu].freelist = kmems[cpu].st_ret[i];
    }
    r = kmems[cpu].freelist;
    kmems[cpu].freelist = r->next;
    release(&kmems[cpu].lock);
  }
  if(r){
    memset((char*)r, 5, PGSIZE); // fill with junk  
    // [generated by LLM] The Chinese string below means "kalloc succeeded".
    DEBUG("kalloc 成功\n");
  }
  pop_off();
  return r;
}

Buffer cache

First, the approach in this section substantially refers to—almost copies—this expert’s blog post.

Lab description

xv6 cannot directly access the disk device. To read disk data, it first copies the data into a cache and then reads the cache.

The smallest unit of disk data in xv6 is one block, whose size is 1024 bytes. In other words, every disk read obtains at least 1024 bytes.

Disk reads and writes call bread() to obtain the appropriate cache buffer, which already contains the data from the corresponding block:

// This file is bio.c
// Return a locked buf with the contents of the indicated block.
struct buf*
bread(uint dev, uint blockno)
{
  struct buf *b;

  b = bget(dev, blockno);
  if(!b->valid) {
    virtio_disk_rw(b, 0);
    b->valid = 1;
  }
  return b;
}

Notice that it first calls bget(). bget() checks whether the disk block is already cached. If it is, it returns the existing buffer. Otherwise, it locates the least recently used buffer and assigns that buffer to the current block:

// Look through buffer cache for block on device dev.
// If not found, allocate a buffer.
// In either case, return locked buffer.
static struct buf*
bget(uint dev, uint blockno)
{
  struct buf *b;

  acquire(&bcache.lock);

  // Is the block already cached?
  for(b = bcache.head.next; b != &bcache.head; b = b->next){
    if(b->dev == dev && b->blockno == blockno){
      b->refcnt++;
      release(&bcache.lock);
      acquiresleep(&b->lock);
      return b;
    }
  }

  // Not cached.
  // Recycle the least recently used (LRU) unused buffer.
  for(b = bcache.head.prev; b != &bcache.head; b = b->prev){
    if(b->refcnt == 0) {
      b->dev = dev;
      b->blockno = blockno;
      b->valid = 0;
      b->refcnt = 1;
      release(&bcache.lock);
      acquiresleep(&b->lock);
      return b;
    }
  }
  panic("bget: no buffers");
}

All buffers are linked into one doubly linked list. The first element is the most recently used and the final element is the least recently used.

Every call to bget() first traverses the list to check whether the current block is cached. If not, it traverses backward from the end, beginning with the least recently used entries, and selects the first buffer whose reference count is zero, meaning no program is using it.

Every cache allocation therefore competes for the lock protecting this list.

The per-core technique from the preceding exercise might seem applicable, but assigning buffers to individual cores does not work well. Allocating or releasing a page frame involves only one core, and an allocated frame is then accessed by a single process.

A buffer cache entry, however, may be accessed by different processes. Several processes can read and write the same cached disk block. If buffers were preassigned by core, a process would frequently need a buffer owned by another core and would have to scan other cores’ caches one by one, reducing performance. Giving every individual buffer its own lock might reduce granularity, but that is a different design.

The lab hint proposes a hash table. It maps block numbers to buckets containing cache buffers. Contention occurs only when two processes operate on buffers in the same bucket. A lookup also traverses only the relevant bucket instead of all cached blocks.

When the corresponding bucket lacks a free buffer, it can steal one from another bucket as kalloc() did.

Approach

The hash table itself is straightforward. However, stealing introduces the same dilemma as page allocation.

During a steal, the code needs the current bucket lock and must also inspect other buckets, requiring their locks. Holding two locks at once can deadlock as follows:^[1]

Assume block b1 hashes to 2 and block b2 hashes to 5,
and neither block is cached before execution.
----------------------------------------
CPU1                  CPU2
----------------------------------------
bget(dev, b1)         bget(dev,b2)
    |                     |
    V                     V
Acquire bucket 2 lock    Acquire bucket 5 lock
    |                     |
    V                     V
Not cached; scan buckets  Not cached; scan buckets
    |                     |
    V                     V
  ......                Reach bucket 2
    |                   Try to acquire bucket 2 lock
    |                     |
    V                     V
  Reach bucket 5       Bucket 2 lock is held by CPU1; wait
Try to acquire bucket 5 lock
    |
    V
Bucket 5 lock is held by CPU2; wait

CPU1 now waits for CPU2 while CPU2 waits for CPU1: deadlock!

One solution is to release the current bucket lock before searching for an unused buffer elsewhere.

This creates a new race. Suppose a process releases its bucket lock and begins searching other buckets for a free buffer. Another process then calls bget() for the same blockno and also begins searching.

After both find free buffers, they may each insert one into the bucket for that block number, leaving two cache entries for the same disk block.

The insertion must therefore be locked, and after acquiring the lock the code must search again for an existing buffer. Another process may have called bget() for the same block concurrently.

Besides locking, we need to identify the least recently used buffer. An LRU buffer is unlikely to be used again soon and is normally recycled when cache space is scarce.

The original design maintained one doubly linked list. A newly released buffer moved to the front, making the tail least recently used.

The new design has several lists, one per bucket, and cannot compare their positions directly. Add lst_use to struct buf to record the last-use time. This time comes from the global ticks variable maintained by timer interrupts:

//trap.c
……

void
clockintr()
{
  acquire(&tickslock);
  ticks++;
  wakeup(&ticks);
  release(&tickslock);
}

if(cpuid() == 0){
  clockintr();
}
……

Code

binit():

#define BUCK_SIZ 13
#define BCACHE_HASH(dev, blk) (((dev << 27) | blk) % BUCK_SIZ) // Support multiple devices;
                                                               // simply taking modulo BUCK_SIZ would also work

// or 13, 1009, 10007
struct {
  struct spinlock bhash_lk[BUCK_SIZ]; // buf hash lock
  struct buf bhash_head[BUCK_SIZ]; // Head of each bucket; avoid buf* because we need the buffer preceding another buffer
                                   // A pointer would make that operation more awkward, as discussed later

  struct buf buf[NBUF]; // The actual cache buffers

  // Linked list of all buffers, through prev/next.
  // Sorted by how recently the buffer was used.
  // head.next is most recent, head.prev is least.
} bcache;

void
binit(void)
{
  for (int i = 0; i < BUCK_SIZ; i++){
    initlock(&bcache.bhash_lk[i], "bcache buf hash lock");
    bcache.bhash_head[i].next = 0;
  }

  for(int i = 0; i < NBUF; i++){ // Initially assign every cache buffer to bucket 0
    struct buf *b = &bcache.buf[i];
    initsleeplock(&b->lock, "buf sleep lock");
    b->lst_use = 0;
    b->refcnt = 0;
    b->next = bcache.bhash_head[0].next; // Insert at the front of bucket 0
    bcache.bhash_head[0].next = b;
  }
}

bget():

This is the primary function being changed.

// Look through buffer cache for block on device dev.
// If not found, allocate a buffer.
// In either case, return locked buffer.
static struct buf*
bget(uint dev, uint blockno)
{
  struct buf *b;

  uint key = BCACHE_HASH(dev, blockno);

  acquire(&bcache.bhash_lk[key]);
  for(b = bcache.bhash_head[key].next; b; b = b->next){
    // Check whether blockno is cached in the corresponding bucket
    if(b->dev == dev && b->blockno == blockno){
      b->refcnt++;
      release(&bcache.bhash_lk[key]);
      acquiresleep(&b->lock);
      return b;
    }
  }
  release(&bcache.bhash_lk[key]);
  int lru_bkt;
  struct buf* pre_lru = bfind_prelru(&lru_bkt);
  // pre_lru returns the address of the list element before the free buffer
  // and ensures that the corresponding bucket lock is held.
  // lru_bkt is an output parameter that receives the buffer's bucket.
  if(pre_lru == 0){
    panic("bget: no buffers");
  }
  
  struct buf* lru = pre_lru->next; 
  // lru, the least recently used buffer with refcnt zero, follows pre_lru
  pre_lru->next = lru->next; 
  // Make pre_lru point to lru's successor, thereby removing lru
  release(&bcache.bhash_lk[lru_bkt]);

  acquire(&bcache.bhash_lk[key]);  

  for(b = bcache.bhash_head[key].next; b; b = b->next){
    // After acquiring the lock, ensure no duplicate buffer has been inserted
    if(b->dev == dev && b->blockno == blockno){
      b->refcnt++;
      release(&bcache.bhash_lk[key]);
      acquiresleep(&b->lock);
      return b;
    }
  }

  lru->next = bcache.bhash_head[key].next; // Add the selected buffer to the front of the list
  bcache.bhash_head[key].next = lru;

  lru->dev = dev, lru->blockno = blockno;
  lru->valid = 0, lru->refcnt = 1; 

  release(&bcache.bhash_lk[key]);

  acquiresleep(&lru->lock);
  return lru;
}

bfind_prelru():

This important helper accepts a pointer to lru_bkt and returns the address of the buffer immediately preceding the least recently used buffer whose reference count is zero. It must continue holding the lock for the bucket containing lru. Otherwise, after releasing that lock and before inserting the buffer into the current bucket, another process could modify the selected lru buffer.

lru_bkt is passed by pointer so the function can assign the bucket number, allowing its caller to know which lock to release.

struct buf* bfind_prelru(int* lru_bkt){ // Return the element before lru while retaining its lock
  struct buf* lru_res = 0;
  *lru_bkt = -1;
  struct buf* b;
  for(int i = 0; i < BUCK_SIZ; i++){
    acquire(&bcache.bhash_lk[i]);
    int found_new = 0;
    for(b = &bcache.bhash_head[i]; b->next; b = b->next){ 
      if(b->next->refcnt == 0 && (!lru_res || b->next->lst_use < lru_res->next->lst_use)){
        lru_res = b;
        found_new = 1;
      }
    }
    if(!found_new){
      // No better choice was found; do not retain this lock because the best bucket lock must remain held
      release(&bcache.bhash_lk[i]);
    }else{ // A better, less recently used choice was found
      if(*lru_bkt != -1) release(&bcache.bhash_lk[*lru_bkt]); // Release the lock for the previous choice
      *lru_bkt = i; // Update the best choice
    }
  }
  return lru_res;
}

brelse():

// Release a locked buffer.
// Move to the head of the most-recently-used list.
void
brelse(struct buf *b)
{
  if(!holdingsleep(&b->lock))
    panic("brelse");

  releasesleep(&b->lock);

  uint key = BCACHE_HASH(b->dev, b->blockno);
  // Obtain the key first after changing to a hash table
  acquire(&bcache.bhash_lk[key]);
  b->refcnt--;
  if (b->refcnt == 0) {
    // no one is waiting for it.
    b->lst_use = ticks;
  }
  
  release(&bcache.bhash_lk[key]);
}

bpin and bunpin:

void
bpin(struct buf *b) {
  uint key = BCACHE_HASH(b->dev, b->blockno);
  acquire(&bcache.bhash_lk[key]);
  b->refcnt++;
  release(&bcache.bhash_lk[key]);
}

void
bunpin(struct buf *b) {
  uint key = BCACHE_HASH(b->dev, b->blockno);
  acquire(&bcache.bhash_lk[key]);
  b->refcnt--;
  release(&bcache.bhash_lk[key]);
}

1.https://blog.miigon.net/posts/s081-lab8-locks/ ↩

Solving Ordinary Differential Equations With the Runge-Kutta Method

2022-08-08T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

[MIT 6.s081] Xv6 Lab 8: Networking Record

2022-08-08T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Update on 2022/9/14: I recently put the lab code on GitHub. If you need a reference, you can find it here:

https://github.com/ttzytt/xv6-riscv

The different branches contain the different labs.

Lab 8: Networking

The lab description is extremely long, although much of it introduces the E1000 network card. The final task is actually simple: implement transmit() and recv() in the E1000 driver.

The code is not complicated, but writing it requires a solid understanding of the hints and consultation of the E1000 documentation.

I will first explain how the processor interacts with the E1000 and then describe the implementation of the two functions.

Interacting with the E1000

The E1000 uses DMA, or direct memory access, and can write received packets directly into computer memory. This is particularly useful for large volumes of data because memory serves as a buffer.

For transmission, software similarly writes descriptors, discussed below, into specific memory locations. The E1000 finds the data awaiting transmission and sends it itself.

For both receiving and transmission, packets are described by arrays of descriptors. The receive and transmit descriptor layouts are introduced in their respective sections.

Receiving

When the network card receives data, it generates an interrupt and invokes the corresponding interrupt handler to process the newly arrived packet.

Descriptors

The receive descriptor format is:

In xv6, it is defined as:

// [E1000 3.2.3]
struct rx_desc
{
  uint64 addr;       /* Address of the descriptor's data buffer */
  uint16 length;     /* Length of data DMAed into data buffer */
  uint16 csum;       /* Packet checksum */
  uint8 status;      /* Descriptor status */
  uint8 errors;      /* Descriptor Errors */
  uint16 special;
};

An array of descriptors is placed in memory and interpreted as a ring queue.

When the card receives a packet, it examines the descriptor at head and writes the data to that descriptor’s buffer, whose address is stored in addr.

The status and length fields are also important, and the card sets both when writing.

length is the size of the packet written at addr. status represents the following states:

The main flag we need is DD, Descriptor Done, which means the card has completely received the packet.

The driver must inspect this flag. If reception is incomplete, it should wait for some additional time.

Ring queue

As described above, the card writes a newly received packet into the buffer of the descriptor at head. We now consider how the card and driver manage this buffer.

The following diagram shows the receive-descriptor ring:

During initialization, head is zero and tail is one less than the queue-buffer size.

The light-colored region from head through tail is free. The diagram appears slightly inaccurate because the position at tail is also free. Software has finished processing every packet in this region. When another packet arrives, the card writes at the beginning of this area, head, overwriting old data, and then increments head.

Software processes the dark region in order. When reading the ring, it reads the buffer at tail + 1, which is the longest-waiting unprocessed packet. After processing the buffer, software increments tail.

Transmission

Descriptors

The transmit descriptor format is:

In xv6, it is defined as:

// [E1000 3.3.3]
struct tx_desc
{
  uint64 addr;
  uint16 length;
  uint8 cso;       // checksum offset
  uint8 cmd;       // command field
  uint8 status;    // 
  uint8 css;       // checksum start field
  uint16 special;  // 
};

addr and length have the same purposes as in a receive descriptor, so I will not repeat them.

The other fields we mainly use are cmd and status.

As on receive descriptors, the DD status flag means that transmission of the referenced data has completed.

cmd describes settings for transmitting the packet—in other words, commands for the network card.

The available commands are:

We need the following commands:

RPS, Report Packet Sent: after this is set, the card reports transmission status. For example, after sending the data referenced by a descriptor, it sets that descriptor’s DD flag.
EOP, End of Packet: indicates that this descriptor is the end of a packet. A very large packet may occupy buffers from several descriptors. EOP is set on its final descriptor. Only then can certain other features, such as IC checksum insertion, be applied.

Ring queue

The transmit-descriptor ring differs slightly from the receive ring. The region from head through tail, shown in a light color, contains data that software wants to send but the card has not yet transmitted.

head points to the longest-waiting pending descriptor, from which the card begins transmission. After finishing, it increments head. New descriptors are inserted at the tail side, and software increments tail as well.

xv6’s representation of network data

To simplify packet handling, xv6 defines struct mbuf:

struct mbuf {
  struct mbuf  *next; // the next mbuf in the chain
  char         *head; // the current start position of the buffer
  unsigned int len;   // the length of the buffer
  char         buf[MBUF_SIZE]; // the backing store
};

e1000_transmit() receives network data in an mbuf, writes it to the memory used by DMA, and thereby lets the card transmit it.

The approximate layout of an mbuf is:

// The above functions manipulate the size and position of the buffer:
//            <- push            <- trim
//             -> pull            -> put
// [-headroom-][------buffer------][-tailroom-]
// |----------------MBUF_SIZE-----------------|
//
// These marcos automatically typecast and determine the size of header structs.
// In most situations you should use these instead of the raw ops above.
#define mbufpullhdr(mbuf, hdr) (typeof(hdr)*)mbufpull(mbuf, sizeof(hdr))
#define mbufpushhdr(mbuf, hdr) (typeof(hdr)*)mbufpush(mbuf, sizeof(hdr))
#define mbufputhdr(mbuf, hdr) (typeof(hdr)*)mbufput(mbuf, sizeof(hdr))
#define mbuftrimhdr(mbuf, hdr) (typeof(hdr)*)mbuftrim(mbuf, sizeof(hdr))----------------MBUF_SIZE-----------------|

Data can be pushed into headroom to store network-protocol headers. After receiving network data, a portion of the central buffer can also be pulled and interpreted as a header structure such as:

// an Ethernet packet header (start of the packet).
struct eth {
  uint8  dhost[ETHADDR_LEN];
  uint8  shost[ETHADDR_LEN];
  uint16 type;
} __attribute__((packed));

The conversion appears in net_rx():

struct eth *ethhdr;
uint16 type;

ethhdr = mbufpullhdr(m, *ethhdr);

The buffer contains the packet body. Tailroom is whatever remains of char buf[MBUF_SIZE] after headroom and the occupied buffer.

Within struct mbuf, len is the length of the body and head marks the end of headroom, or the current beginning of the buffer.

net.c contains many mbuf-related functions. The most important are mbufalloc() and mbuffree(), which allocate and release an mbuf.

Register operations

Specific memory mappings provide access to the E1000 control registers. More precisely, code adds offsets to the global regs variable in e1000.c; those offsets are defined in e1000_dev.h.

Implementation and explanation

Transmission

The overall idea follows the lab hints.

First, read the current ring tail—the first descriptor not currently being transmitted—from the memory-mapped control register, and obtain its descriptor:

1
2
3

acquire(&e1000_lock); // Multiple threads may transmit simultaneously, so acquire the lock
uint idx = regs[E1000_TDT]; // Transmit tail, identifying the first free ring descriptor
struct tx_desc *desc = &tx_ring[idx];

Then inspect the descriptor status. If E1000_TXD_STAT_DD is clear, the ring has no free position: tail has reached the light-colored region because the entire queue contains pending transmit descriptors. Return immediately in this case.

if(!(desc->status & E1000_TXD_STAT_DD)){ // If transmission is incomplete, the ring has no free buffer
  release(&e1000_lock);
  return -1;
}

Next, check the mbuf associated with this descriptor. Its addr points to the mbuf. If the descriptor’s old data has finished transmitting, release that mbuf.

if(tx_mbufs[idx] != NULL){ // This buffer points to the packet being transmitted
  // The preceding check guarantees that transmission has completed.
  // tx_mbufs requires no allocation and points directly to argument m.
  mbuffree(tx_mbufs[idx]);
  tx_mbufs[idx] = NULL;
}

After freeing the old buffer, point addr at the data currently being sent and update the length:

1 2	desc->addr = m->head; desc->length = m->len;

It took me a long time to understand why the assignment is desc->addr = m->head rather than desc->addr = m->buf.

I initially thought an mbuf’s headroom stored the packet header. In fact, the header is stored at the beginning of the central buffer, while headroom is only reserve space. If the current header must be replaced by a larger one, for example, code can call mbufpullhdr() and then mbufpushhdr().

An example caller of e1000_transmit() shows the purpose of headroom. The only caller in net.c is net_tx_eth():

// sends an ethernet packet
static void
net_tx_eth(struct mbuf *m, uint16 ethtype)
{
  struct eth *ethhdr;
  ethhdr = mbufpushhdr(m, *ethhdr); // Notice this line
  memmove(ethhdr->shost, local_mac, ETHADDR_LEN);
  // In a real networking stack, dhost would be set to the address discovered
  // through ARP. Because we don't support enough of the ARP protocol, set it
  // to broadcast instead.
  memmove(ethhdr->dhost, broadcast_mac, ETHADDR_LEN);
  ethhdr->type = htons(ethtype);
  if (e1000_transmit(m)) {
    mbuffree(m);
  }
}

This function primarily adds an Ethernet header. ethhdr = mbufpushhdr(m, *ethhdr); shrinks headroom and enlarges the buffer, returning the newly added space as ethhdr.

The following calls to memmove(ethhdr->shost, local_mac, ETHADDR_LEN) and memmove(ethhdr->dhost, broadcast_mac, ETHADDR_LEN) copy the header fields into that space carved out of headroom. The mbuf’s buffer now includes the packet header.

If a larger header is later required, headroom can again be reduced to enlarge the buffer.

Returning to e1000_transmit(), after setting addr and length, set the descriptor commands:

1	desc->cmd = E1000_TXD_CMD_RS \| E1000_TXD_CMD_EOP;

The two commands were explained in the transmit-descriptor section above.

The final code in e1000_transmit() is:

tx_mbufs[idx] = m; // Record it for later cleanup

regs[E1000_TDT] = (idx + 1) % TX_RING_SIZE; // Update tail

release(&e1000_lock);
return 0;

The main line to explain is tx_mbufs[idx] = m. Earlier, the function checked E1000_TXD_STAT_DD to learn whether transmission of this descriptor had finished. If not, it returned. If so, it released the old packet buffer.

Assigning m to tx_mbufs[idx] records the buffer so the next use of this descriptor can inspect and clean up its transmission state.

The complete e1000_transmit() is:

int
e1000_transmit(struct mbuf *m)
{
  acquire(&e1000_lock);
  uint idx = regs[E1000_TDT];
  struct tx_desc *desc = &tx_ring[idx];
  if(!(desc->status & E1000_TXD_STAT_DD)){
    release(&e1000_lock);
    return -1;
  }

  if(tx_mbufs[idx] != NULL){
    mbuffree(tx_mbufs[idx]);
    tx_mbufs[idx] = NULL;
  }

  desc->addr = m->head;
  desc->length = m->len;

  desc->cmd = E1000_TXD_CMD_RS | E1000_TXD_CMD_EOP;
  
  tx_mbufs[idx] = m; 

  regs[E1000_TDT] = (idx + 1) % TX_RING_SIZE;

  release(&e1000_lock);
  return 0;
}

Receiving

One important point is that e1000_recv() must read every currently pending packet in one invocation. It therefore needs a loop that repeatedly reads the descriptor at tail until it finds one whose reception is incomplete.

The E1000 supports several interrupt strategies for received packets. A common one is RDTR, the Receive Interrupt Delay Timer. Roughly, after a packet is received and DMA writes it into host memory, a timer begins; the interrupt occurs only after the configured delay.

This strategy reduces interrupt volume when many packets arrive in a short interval. xv6 does not use it, however, and instead generates an interrupt after every write into host memory:

1 2	regs[E1000_RDTR] = 0; // interrupt after every received packet (no timer) regs[E1000_RADV] = 0; // interrupt after every packet (no timer)

If every packet produces an interrupt, why not read only one descriptor per interrupt instead of looping at tail?

My understanding is that interrupt delivery is disabled while handling an external-device interrupt.

Suppose many packets arrive quickly. The first interrupt begins handling, but several more interrupts may be generated before that handler finishes. Those later interrupts cannot be received while interrupts are disabled, especially if processing a descriptor is slower than packet arrival.

The handler therefore checks for additional arrived packets every time it runs and continues reading while they exist.

Returning to the implementation, first read tail and obtain its descriptor:

1 2	uint idx = (regs[E1000_RDT] + 1) % RX_RING_SIZE; // The region from head through tail is free struct rx_desc *desc = &rx_ring[idx];

Tail itself is a free buffer whose data was processed earlier, so increment tail before selecting the next descriptor.

Use the DD flag to determine whether all pending descriptors have been read:

1
2
3

if(!(desc->status & E1000_RXD_STAT_DD)){
  return;
}

Set the mbuf length from the received descriptor:

1	rx_mbufs[idx]->len = desc->length;

Unlike transmission, each receive descriptor has a permanently associated mbuf. Its addr was initialized beforehand. The initialization code allocates the first set of mbufs as follows:

// [E1000 14.4] Receive initialization
memset(rx_ring, 0, sizeof(rx_ring));
  for (i = 0; i < RX_RING_SIZE; i++) {
  rx_mbufs[i] = mbufalloc(0);
  if (!rx_mbufs[i])
      panic("e1000");
  rx_ring[i].addr = (uint64) rx_mbufs[i]->head;
}

Then pass the mbuf to net_rx() for processing by the appropriate network-protocol stack:

1	net_rx(rx_mbufs[idx]);

The upper protocol layers still need the old mbuf, so it cannot be overwritten. Allocate a fresh mbuf for the current descriptor:

1
2
3

rx_mbufs[idx] = mbufalloc(0);
desc->addr = rx_mbufs[idx]->head;
desc->status = 0;

Finally, update tail. Remember that tail itself points to a descriptor already processed by software:

1	regs[E1000_RDT] = idx;

The complete e1000_recv() is:

static void
e1000_recv(void)
{
  while(1){
    uint idx = (regs[E1000_RDT] + 1) % RX_RING_SIZE;
    struct rx_desc *desc = &rx_ring[idx];
    if(!(desc->status & E1000_RXD_STAT_DD)){
      return;
    } 
    rx_mbufs[idx]->len = desc->length;
    net_rx(rx_mbufs[idx]);
    rx_mbufs[idx] = mbufalloc(0);
    desc->addr = rx_mbufs[idx]->head;
    desc->status = 0;
    regs[E1000_RDT] = idx;
  }
}

After completing these functions, the lab passes:

[MIT 6.s081] Xv6 Lab 7: Multithreading Record

2022-08-04T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Update on 2022/9/14: I recently put the lab code on GitHub. If you need a reference, you can find it here:

https://github.com/ttzytt/xv6-riscv

The different branches contain the different labs.

Lab 7: Multithreading

Uthread

Implement user-mode threads.

Because we are implementing user-mode multithreading, much of the design can follow the kernel’s multithreading implementation.

Inspection of user/uthread.c shows that the basic framework has already been provided. We only need to implement several functions.

First, clarify what each unfinished function should do:

thread_switch() is identical to the kernel’s swtch() and switches processor context. As in the kernel implementation discussed in this article, this switch occurs through a normal function call, so caller-saved registers do not need to be saved and exchanged.
thread_create() creates a new user thread. Following the kernel implementation, after swtch() the ra register determines the destination and sp determines the restored callee-saved-register context. Set ra so that the first execution of the user thread begins at the first instruction of its function.
thread_schedule() plays the same role as the kernel’s scheduler(). After the current thread calls yield(), it finds a RUNNABLE thread and executes it. thread_schedule() calls thread_switch() to exchange processor context.

With the purpose of each function clear, we can begin the implementation.

The original uthread.c does not include a context field in struct thread, so add one. The saved registers are exactly the same as those used by kernel-mode threads:

struct Context{
  uint64 ra;
  uint64 sp;

  // callee-saved
  uint64 s0;
  uint64 s1;
  uint64 s2;
  uint64 s3;
  uint64 s4;
  uint64 s5;
  uint64 s6;
  uint64 s7;
  uint64 s8;
  uint64 s9;
  uint64 s10;
  uint64 s11;
};

struct thread {
  char       stack[STACK_SIZE]; /* the thread's stack */
  int        state;             /* FREE, RUNNING, RUNNABLE */
  struct Context ctx;
};

thread_switch() can then copy the contents of swtch() almost directly:

 .text

 /*
         * save the old thread's registers,
         * restore the new thread's registers.
         */

 .globl thread_switch
 // a0 is the old context and a1 is the new context
thread_switch:
 /* YOUR CODE HERE */
 sd ra, 0(a0)
 sd sp, 8(a0)
 sd s0, 16(a0)
 sd s1, 24(a0)
 sd s2, 32(a0)
 sd s3, 40(a0)
 sd s4, 48(a0)
 sd s5, 56(a0)
 sd s6, 64(a0)
 sd s7, 72(a0)
 sd s8, 80(a0)
 sd s9, 88(a0)
 sd s10, 96(a0)
 sd s11, 104(a0)

 ld ra, 0(a1)
 ld sp, 8(a1)
 ld s0, 16(a1)
 ld s1, 24(a1)
 ld s2, 32(a1)
 ld s3, 40(a1)
 ld s4, 48(a1)
 ld s5, 56(a1)
 ld s6, 64(a1)
 ld s7, 72(a1)
 ld s8, 80(a1)
 ld s9, 88(a1)
 ld s10, 96(a1)
 ld s11, 104(a1)

 ret    /* return to ra */

That completes the first function.

Next is thread_create(). Its main challenge is setting ra and sp correctly. A new user thread has not used its registers yet, so the initial values of the other context registers do not matter.

After thread_create(), a call to thread_schedule() should execute the first statement of the thread function. Set ra as follows:

1	t->ctx.ra = (uint64) func;

For sp, remember that the stack grows from high addresses toward low addresses—something I initially forgot. Set sp to the highest address in the stack:

1	t->ctx.sp = (uint64) &t->stack + (STACK_SIZE - 1);

The complete thread_create() is therefore:

void 
thread_create(void (*func)())
{
  struct thread *t;

  for (t = all_thread; t < all_thread + MAX_THREAD; t++) {
    if (t->state == FREE) break;
  }
  t->state = RUNNABLE;
  // YOUR CODE HERE
  t->ctx.ra = (uint64) func;
  t->ctx.sp = (uint64) &t->stack + (STACK_SIZE - 1);
}

We can now handle thread_schedule().

The original loop finds the first RUNNABLE thread and assigns it to next_thread. Clearly, the contexts of current_thread and next_thread should be exchanged.

One slightly tricky detail is that the function performs the following assignment before the switch:

1 2	t = current_thread; current_thread = next_thread; // The current thread becomes the next thread

We therefore switch between t and next_thread:

1	thread_switch((uint64) &t->ctx, (uint64) &next_thread->ctx);

The complete code is:

void 
thread_schedule(void)
{
 struct thread *t, *next_thread;

  /* Find another runnable thread. */
  next_thread = 0;
  t = current_thread + 1;
  for(int i = 0; i < MAX_THREAD; i++){
    if(t >= all_thread + MAX_THREAD)
      t = all_thread; // Wrap around
    if(t->state == RUNNABLE) {
      next_thread = t;
      break;
    }
    t = t + 1;
  }

  if (next_thread == 0) {
    printf("thread_schedule: no runnable threads\n");
    exit(-1);
  }

  if (current_thread != next_thread) {         /* switch threads?  */
    next_thread->state = RUNNING;
    t = current_thread;
    current_thread = next_thread; // The current thread becomes the next thread
    /* YOUR CODE HERE
     * Invoke thread_switch to switch from t to next_thread:
     * thread_switch(??, ??);
     */
    thread_switch((uint64) &t->ctx, (uint64) &next_thread->ctx);
  } else
    next_thread = 0;
}

After reading other people’s blog posts,^[1] I realized that the user-mode multithreading implemented here is actually closer to coroutines. Threads voluntarily yield processor resources instead of being preempted by timer interrupts, and only one core is used.

In other words, a function can suspend itself and later resume through thread_schedule().

I had previously read about coroutines and could understand only why they were called “functions that can be suspended.” I could not understand why a coroutine was considered a “user-mode thread,” much less how one was implemented.

It is a strange and satisfying experience to study one topic and suddenly understand something apparently unrelated that had remained incomprehensible before. If learning something has stalled for a long time, perhaps set it aside; at some unexpected time later, another topic may make it clear.

Using threads

The lab description is long, so I will not reproduce its image here. The task is roughly to read a hash-table program and modify it so that it also works in a multithreaded environment.

Running the supplied program with one thread works normally. With two or more threads, some key-value pairs inserted into the hash table disappear entirely.

To solve the problem, inspect the hash table and locate the race. Its three most important functions are insert(), put(), and get().

First, insert():

static void 
insert(int key, int value, struct entry **p, struct entry *n)
{
  struct entry *e = malloc(sizeof(struct entry));
  e->key = key;
  e->value = value;
  e->next = n;
  *p = e; // Change the head of p, table[i], to e
}

In a hash table, when the hash function maps several different keys to the same location, that bucket is represented as a linked list. A lookup traverses the list to find the correct key-value pair.

insert() adds an element to such a list. e is the new element inserted into list *p; its fields are initialized from the arguments.

The expression e->next = n is especially important. Here, n is the first element of table[i], or *p. Assigning it to e->next inserts e before the previous head.

The next function is put():

static 
void put(int key, int value)
{
  // is the key already present?
  struct entry *e = 0;
  for (e = table[i]; e != 0; e = e->next) {
    if (e->key == key)
      break;
  }
  if(e){
    // update the existing key.
    e->value = value;
  } else {
    // the new is new.
    insert(key, value, &table[i], table[i]); // Insert a key-value pair at the front of table[i]
  }
}

It attempts to add a key-value pair to the hash table. First, it searches for key. If that key already exists, its old value is replaced with value.

Otherwise, it calls insert() to add the pair.

The final important function is get():

static struct entry*
get(int key)
{
  int i = key % NBUCKET;
  struct entry *e = 0;
  for (e = table[i]; e != 0; e = e->next) {
    if (e->key == key) break;
  }
  return e;
}

It traverses the appropriate linked list in the hash table to find the value associated with the key.

Overall, this is an ordinary hash-table implementation and appears correct, but it contains a race in a multithreaded environment.

Consider the following situation.^[1]

Keys k1 and k2 belong to the same bucket list, and neither pair is currently present. Threads t1 and t2 attempt to insert the two keys into that list.

t1 first verifies that k1 is absent and prepares to call insert() to add it at the front.

At this moment, the scheduler switches to t2. Alternatively, on a multicore machine both run in parallel and t2 advances more quickly.

t2 also observes that k2 is absent and calls insert(). After insertion, k2 is the first element.

t1 then performs its insertion of k1. Because it does not know that t2 has inserted k2 at the head, it writes k1 using the head that it previously observed. k2 is overwritten, and the key-value pair is lost.

This race requires locking.

The preceding scenario shows that at any instant, only one thread may operate on a particular hash-table bucket, including both reads and modifications. Multiple threads may otherwise observe stale information, as above.

Create one mutex for every bucket list, and lock and unlock it at the beginning and end of put() and get().

Why not lock inside insert()? Because insert() is called by put(), and acquiring the same nonrecursive mutex twice would deadlock.

The modified functions are:

pthread_mutex_t bkt_lock[NBUCKET];

static 
void put(int key, int value)
{
  int i = key % NBUCKET;
  
  pthread_mutex_lock(&bkt_lock[i]);
  // is the key already present?
  struct entry *e = 0;
  for (e = table[i]; e != 0; e = e->next) {
    if (e->key == key)
      break;
  }
  if(e){
    // update the existing key.
    e->value = value;
  } else {
    // the new is new.
    insert(key, value, &table[i], table[i]); // Insert a key-value pair at the front of table[i]
  }
  pthread_mutex_unlock(&bkt_lock[i]);
}

static struct entry*
get(int key)
{
  int i = key % NBUCKET;

  pthread_mutex_lock(&bkt_lock[i]);
  
  struct entry *e = 0;
  for (e = table[i]; e != 0; e = e->next) {
    if (e->key == key) break;
  }
  pthread_mutex_unlock(&bkt_lock[i]);
  return e;
}

Barrier

Implement a synchronization barrier.

First, what is a synchronization barrier? According to Wikipedia:

A barrier is a synchronization method in parallel computing. For a group of processes or threads, a barrier means that any thread or process reaching that point must wait until all of them have arrived before execution can continue.

A naive implementation increments a variable whenever a thread reaches the barrier. Once the value equals the total thread count, execution may continue.

Before the count reaches the total, each thread must block at the barrier. When the condition becomes true, all blocked threads may cross it.

We could combine a mutex with polling to check the variable repeatedly, but this would have a significant performance cost.

Polling is passive: every thread repeatedly asks whether the condition is true. Why not instead let the final arriving thread notify all the others?

Condition variables in the pthread library provide exactly this function.

For example, after calling pthread_cond_wait(&cond, &mutex), a thread remains blocked until another thread calls pthread_cond_broadcast(&cond).

More specifically, pthread_cond_wait(&cond, &mutex) performs these operations in order:

Calls pthread_mutex_unlock(&mutex).
Places the thread in the list of threads waiting for the condition.
Blocks the thread until another thread sends a signal.

Steps 1 and 2 are atomic.

When a thread signals the condition variable:

The kernel wakes waiting threads; the number depends on whether signal or broadcast was used.
pthread_cond_wait() returns in each awakened thread.
mutex is locked again.

Why must a condition variable be paired with a mutex? Here is my present understanding.

A condition variable is normally used together with another variable, which I will call x.

Before calling wait(), a thread checks whether x satisfies a condition. If it already does, there is no need to wait.

If not, the thread calls wait() so that it can be notified when x later satisfies the condition.

x is shared in a multithreaded environment. The thread must therefore hold a lock protecting x while checking it before the wait.

After discovering that x does not satisfy the condition and entering wait(), the lock must be released. Otherwise, no other thread can modify x to make the condition true.

Likewise, when x becomes suitable and wait() returns, the awakened thread reacquires the lock protecting x. It may inspect or modify x, and a concurrent change at that point would be unsafe.

Why must unlocking and joining the wait queue be one atomic operation?

Suppose a program uses a condition variable but performs those operations separately:^[2]

lock(x_lock) // Acquire the lock protecting x
if (x satisfies the condition){
    unlock(x_lock); // Release the lock protecting x
    pthread_cond_wait(&cond); // Wait for a signal
    lock(x_lock); // dosomething may modify x
    dosomething();
}
unlock();

Between unlock(x_lock) and placing the current thread in the wait queue for cond, another thread might change x and signal the condition. Because the current thread has not yet joined the queue, it misses the signal permanently.

Therefore, queue insertion and unlocking must be atomic.

I did not expect to write so much about condition variables while barely mentioning barriers. Now return to the actual task and implement the barrier.

First inspect the supplied barrier structure in barrier.c:

struct barrier {
  pthread_mutex_t barrier_mutex;
  pthread_cond_t barrier_cond;
  int nthread;      // Number of threads that have reached this round of the barrier
  int round;     // Barrier round
} bstate;

nthread is the variable x described above: threads wait on the condition variable only while nthread has not reached the required value.

The mutex that protects x is correspondingly barrier_mutex. This gives the following implementation:

static void 
barrier()
{
  // YOUR CODE HERE
  //
  // Block until all threads have called barrier() and
  // then increment bstate.round.
  //
  pthread_mutex_lock(&bstate.barrier_mutex);
  bstate.nthread++;
  if(bstate.nthread < nthread){
    pthread_cond_wait(&bstate.barrier_cond, &bstate.barrier_mutex);
    // Wait if not every thread has reached the barrier.
    // This call blocks until a signal is received.
  }else{ // This is the final thread.
    bstate.nthread = 0;
    bstate.round++;
    pthread_cond_broadcast(&bstate.barrier_cond);
  }
  pthread_mutex_unlock(&bstate.barrier_mutex);
}

One detail is the distinction between pthread_cond_broadcast() and pthread_cond_signal().

broadcast() wakes every thread in the waiting list, whereas signal() wakes only one.

When the final thread reaches this barrier, all threads may proceed, so the implementation uses broadcast().

The lab now passes. I wish everyone working on it an early AC:

Summary

Writing a blog is important. Producing working code does not necessarily mean that I understand it completely. With the condition variable in the final exercise, for example, I understood what it did and felt that the implementation was fine. While writing the article, however, I discovered that I could not explain it and had to consult more material. Explaining knowledge to someone else requires a deeper understanding of it.

The amount of code in this lab is relatively small. To be honest, none of the labs so far has required especially much code. It is possible to finish without fully understanding scheduling and context switching in xv6. Completing it after understanding those mechanisms, however, gives much more insight—especially for the uthread exercise, since the other two depend more heavily on pthread.

[MIT 6.s081] Xv6 Lab 6: Copy-on-Write Record

2022-07-29T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Update on 2022/9/14: I recently put the lab code on GitHub. If you need a reference, you can find it here:

https://github.com/ttzytt/xv6-riscv

The different branches contain the different labs.

Lab 6: Copy-on-Write Fork for xv6

The lab description itself is remarkably brief because its main explanation appears immediately beforehand:

The problem
The fork() system call in xv6 copies all of the parent process’s user-space memory into the child. If the parent is large, copying can take a long time. Worse, the work is often largely wasted; for example, a fork() followed by exec() in the child will cause the child to discard the copied memory, probably without ever using most of it. On the other hand, if both parent and child use a page, and one or both writes it, a copy is truly needed.
The solution
The goal of copy-on-write (COW) fork() is to defer allocating and copying physical memory pages for the child until the copies are actually needed, if ever.
COW fork() creates just a pagetable for the child, with PTEs for user memory pointing to the parent’s physical pages. COW fork() marks all the user PTEs in both parent and child as not writable. When either process tries to write one of these COW pages, the CPU will force a page fault. The kernel page-fault handler detects this case, allocates a page of physical memory for the faulting process, copies the original page into the new page, and modifies the relevant PTE in the faulting process to refer to the new page, this time with the PTE marked writeable. When the page fault handler returns, the user process will be able to write its copy of the page.
COW fork() makes freeing of the physical pages that implement user memory a little trickier. A given physical page may be referred to by multiple processes’ page tables, and should be freed only when the last reference disappears.

In short, we need to implement the copy-on-write technique used by UNIX. Without COW, fork() copies all memory belonging to the parent into the child’s address space. This consumes an enormous and unacceptable amount of time for a large process.

Much of the memory copied during fork() is never used. For example, the usual UNIX process-creation sequence first invokes fork() and then exec(). All data copied from the parent is immediately discarded by exec().

There is only one situation in which memory truly needs to be copied during a fork: a write. If the parent or child attempts to write a value at an address, the corresponding page frame must be copied so that the write does not change memory observed by the other process.

Copy-on-write implements precisely this behavior. Mark shared parent and child PTEs as not writable. When either process attempts to write a shared page, the CPU generates a page fault. usertrap() handles it by copying the shared frame for the writing process and marking that new frame writable.

After COW is implemented, several processes may share one physical page frame. That frame can be truly released only after every process has stopped using it.

We can now follow the hints one at a time.

uvmcopy()

Modify uvmcopy() to map the parent’s physical pages into the child, instead of allocating new pages. Clear PTE_W in the PTEs of both child and parent.
Modify uvmcopy() so that the parent’s physical memory is mapped directly into the child’s virtual address space rather than allocating new memory. Clear PTE_W in both parent and child PTEs.

After this change, the parent and child effectively share memory. We want a write by either process to cause a page fault, so PTE_W must be cleared:

// Given a parent process's page table, copy
// its memory into a child's page table.
// Copies both the page table and the
// physical memory.
// returns 0 on success, -1 on failure.
// frees any allocated pages on failure.
int
uvmcopy(pagetable_t old, pagetable_t new, uint64 sz)
{
  pte_t *pte;
  uint64 pa, i;
  uint flags;
  char *mem;

  for(i = 0; i < sz; i += PGSIZE){
    if((pte = walk(old, i, 0)) == 0)
      panic("uvmcopy: pte should exist");
    if((*pte & PTE_V) == 0)
      panic("uvmcopy: page not present");
    pa = PTE2PA(*pte);

    *pte &= (~PTE_W); // Clear PTE_W here
    *pte |= PTE_C;    // PTE_C marks this as a COW page, as discussed later
    flags = PTE_FLAGS(*pte);
    // if((mem = kalloc()) == 0)  These lines allocate physical memory and must be removed
    //   goto err;
    // memmove(mem, (char*)pa, PGSIZE);
    if(mappages(new, i, PGSIZE, (uint64)pa, flags) != 0){ 
      // Do not map virtual address i to newly allocated physical memory mem.
      // Map it to the parent's physical memory at pa instead.
      printf("uvmcopy failed\n");
      kfree(mem);
      goto err;
    }
    refcnt_inc(pa); // Discussed later
  }
  return 0;

 err:
  uvmunmap(new, 0, i / PGSIZE, 1);
  return -1;
}

usertrap()

Modify usertrap() to recognize page faults. When a page-fault occurs on a COW page, allocate a new page with kalloc(), copy the old page to the new page, and install the new page in the PTE with PTE_W set.
Modify usertrap() to handle page faults. When the fault occurs on a COW page, allocate a new physical page, copy the original frame into it, and install the new page with PTE_W set.

As in the lazy-allocation lab, we need a helper that determines whether a virtual address is a valid, not-yet-copied COW page. The hint says that a new physical page may be allocated only when the fault occurs on a COW page. How can we distinguish one? We can use the reserved bits in a RISC-V PTE. Each PTE has ten flag bits; eight have defined meanings, leaving bits 8 and 9 reserved:

These RSW bits are available to software.

Define bit 8 as an indicator that the frame is a COW page. Add the following macro in kernel/riscv.h. This also explains why uvmcopy() above sets PTE_C in the child PTE:

#define PTE_V (1L << 0) // valid
#define PTE_R (1L << 1)
#define PTE_W (1L << 2)
#define PTE_X (1L << 3)
#define PTE_U (1L << 4) // 1 -> user can access
#define PTE_C (1L << 8) // Newly added

The helper that detects an uncopied COW page follows. As in the lazy-allocation lab, I placed it in vm.c:

int uncopied_cow(pagetable_t pgtbl, uint64 va){
  if(va >= MAXVA) 
    return 0;
  pte_t* pte = walk(pgtbl, va, 0);
  if(pte == 0)             // This page does not exist
    return 0;
  if((*pte & PTE_V) == 0)
    return 0;
  if((*pte & PTE_U) == 0)
    return 0;
  return ((*pte) & PTE_C); // PTE_C means this is an uncopied COW page
}

We can now modify usertrap():

……
  syscall();
  } else if((which_dev = devintr()) != 0){
    // ok
  } else if(r_scause() == 15 && uncopied_cow(p->pagetable, r_stval())){ 
    if(cowalloc(p->pagetable, r_stval()) < 0){
      p->killed = 1;
    }
  } else {
    printf("usertrap(): unexpected scause %p pid=%d\n", r_scause(), p->pid);
    printf("            sepc=%p stval=%p\n", r_sepc(), r_stval());
    p->killed = 1;
  }
……

Unlike the lazy-allocation lab, only scause 15 is handled. According to the RISC-V documentation:

An scause of 15 means a page fault caused by an attempted write.

After determining that the current page is a valid COW page, allocate physical memory for it. As in the preceding lab, I wrapped this work in cowalloc():

int cowalloc(pagetable_t pgtbl, uint64 va){
  pte_t* pte = walk(pgtbl, va, 0);
  uint64 perm = PTE_FLAGS(*pte);

  if(pte == 0) return -1;
  uint64 prev_sta = PTE2PA(*pte); // prev_sta is the parent's page frame originally used by this mapping.
                                  // The name uses sta because the address is page-aligned
                                  // and therefore denotes the start of a page frame.
  uint64 newpage = kalloc();     
  if(!newpage){
    return -1;
  }
  uint64 va_sta = PGROUNDDOWN(va); // Current page frame

  perm &= (~PTE_C); // After copying, it is no longer an uncopied COW page
  perm |= PTE_W;    // It becomes writable after copying

  memmove(newpage, prev_sta, PGSIZE); // Copy the parent's page-frame data
  uvmunmap(pgtbl, va_sta, 1, 1);      // Then remove the mapping to the parent's frame
  
  if(mappages(pgtbl, va_sta, PGSIZE, (uint64)newpage, perm) < 0){
    kfree(newpage);
    return -1;
  }
  return 0;
}

One detail is critical: memmove() must precede uvmunmap(). This took me a long time to debug. After uvmunmap(), the parent’s physical page may have been released, so a later memmove() would read invalid data.

After reading this function, another problem may be apparent. The parent’s page frame might be shared by more than one child. Calling uvmunmap() with do_free equal to one may release the parent frame while other processes still use it.

This leads to the next hint.

Reference count

Ensure that each physical page is freed when the last PTE reference to it goes away – but not before. A good way to do this is to keep, for each physical page, a “reference count” of the number of user page tables that refer to that page. Set a page’s reference count to one when kalloc() allocates it. Increment a page’s reference count when fork causes a child to share the page, and decrement a page’s count each time any process drops the page from its page table. kfree() should only place a page back on the free list if its reference count is zero. It’s OK to keep these counts in a fixed-size array of integers. You’ll have to work out a scheme for how to index the array and how to choose its size. For example, you could index the array with the page’s physical address divided by 4096, and give the array a number of elements equal to highest physical address of any page placed on the free list by kinit() in kalloc.c.

We need reference counts to solve the problem. Every page frame has a count recording how many COW mappings refer to it. Only when no COW page still uses a frame can it actually be freed, somewhat like the behavior of close(). kalloc() sets the reference count of a newly allocated page to one. kfree() first decrements the count and frees the frame only when the result reaches zero.

We also need a way to store these counts. Because every page-frame start address is divisible by 4096, divide the physical address of a frame by 4096 to obtain its index.

This gives the following macros and array:

#define PG2REFIDX(_pa) ((((uint64)_pa) - KERNBASE) / PGSIZE)
#define MX_PGIDX PG2REFIDX(PHYSTOP)
#define PG_REFCNT(_pa) pg_refcnt[PG2REFIDX((_pa))]

int pg_refcnt[MX_PGIDX];

The following diagram helps explain the calculation:

PHYSTOP and KERNBASE mark the beginning and end of the physical-memory range, so subtract KERNBASE from pa before dividing by PGSIZE.

At first I wondered where this array would itself be stored in kernel memory. The implementation of kinit() gives the answer:

void
kinit()
{
  initlock(&kmem.lock, "kmem");
  freerange(end, (void*)PHYSTOP); // Notice this line
}

Here, end is the beginning of Free memory in the diagram and is defined in kernel.ld. The kernel’s own code and data, including this array, live in kernel text and kernel data. kalloc() allocates only from the range end through PHYSTOP.

We can now modify the functions in kalloc.c around the reference counts.

First, kalloc():

void *
kalloc(void)
{
  struct run *r;

  acquire(&kmem.lock);
  r = kmem.freelist;
  if(r){
    kmem.freelist = r->next;
  }
  release(&kmem.lock);

  if(r){
    memset((char*)r, 5, PGSIZE); // fill with junk
    PG_REFCNT(r) = 1;            
    // One process uses the newly allocated frame, so initialize the count to one.
  }
  return (void*)r;
}

Next, kfree():

void
kfree(void *pa)
{
  struct run *r;

  if(((uint64)pa % PGSIZE) != 0 || (char*)pa < end || (uint64)pa >= PHYSTOP)
    panic("kfree");

  acquire(&refcnt_lock);
  if(--PG_REFCNT(pa) <= 0){ // Decrement first; truly free the frame only at zero
    memset(pa, 1, PGSIZE);
    // Fill with junk to catch dangling refs.
    r = (struct run*)pa;
    acquire(&kmem.lock);
    r->next = kmem.freelist;
    kmem.freelist = r;
    release(&kmem.lock);
  }
  release(&refcnt_lock);
}

refcnt_lock is initialized in kinit():

void
kinit()
{
  initlock(&kmem.lock, "kmem");
  initlock(&refcnt_lock, "ref cnt"); // here
  freerange(end, (void*)PHYSTOP);
}

The lock is required because several processes referencing the same frame may call kfree() simultaneously. Concurrent decrements without synchronization could produce an incorrect result.

In uvmcopy(), increment the reference count for the parent’s page frame because one additional process now shares it. This is the refcnt_inc() call after mappages(), whose definition is:

void refcnt_inc(void* pa){
  acquire(&refcnt_lock);
  PG_REFCNT(pa)++;
  release(&refcnt_lock);
}

This completes the reference-counting portion.

One final hint remains.

copyout()

The reason for modifying copyout() resembles the previous lab. Some system calls write data into COW pages. Because PTE_W is clear on those pages, the write generates a page fault. trap.c treats an exception occurring inside a system call as a kernel fault and panics. Therefore, if copyout() discovers a COW page, it should allocate a private page directly.

Unlike the lazy-allocation lab, copyin does not need a modification here. copyin() can read the physical frame shared by the parent. In the lazy-allocation lab, by contrast, the frame being read had no physical address at all and therefore could not be accessed.

Modify copyout() as follows:

// Copy from kernel to user.
// Copy len bytes from src to virtual address dstva in a given page table.
// Return 0 on success, -1 on error.
int
copyout(pagetable_t pagetable, uint64 dstva, char *src, uint64 len)
{
  uint64 n, va0, pa0;

  while(len > 0){
    va0 = PGROUNDDOWN(dstva); 
    if(uncopied_cow(pagetable, va0)){          // Newly added
      try(cowalloc(pagetable, va0), return -1);
    }
    pa0 = walkaddr(pagetable, va0);
    if(pa0 == 0)
      return -1;
    n = PGSIZE - (dstva - va0);
    if(n > len)
      n = len;
    memmove((void *)(pa0 + (dstva - va0)), src, n);

    len -= n;
    src += n;
    dstva = va0 + PGSIZE;
  }
  return 0;
}

When writing this function, the order of cowalloc() and walkaddr() is crucial. I originally reversed them and spent a long time finding the problem. Calling walkaddr() before cowalloc() returns the physical address of the parent’s shared frame.

The later write would then modify that shared address, corrupting data used by other processes.

Calling walkaddr() after cowalloc() instead returns the newly allocated physical address. The write goes to a frame owned by the current process and cannot affect any other process.

With this function complete, the lab passes. I wish everyone working on it an early AC:

Summary

I cannot understand why GDB failed to reveal several foolish mistakes even after so much debugging. I began to suspect the compiler itself. In the future, I need to reason through the design before writing it. If the implementation is wrong and my debugging assumptions follow the same wrong direction, the bug will never be found.

[MIT 6.s081] Xv6 Lab 5 (2020): Lazy Page Allocation Record

2022-07-28T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Update on 2022/9/14: I recently put the lab code on GitHub. If you need a reference, you can find it here:

https://github.com/ttzytt/xv6-riscv

The different branches contain the different labs.

Lab 5 (2020): lazy page allocation

Eliminate allocation from sbrk()

Remove the part of the sbrk() system call that actually allocates memory.

There is not much to explain here. Follow the hint and remove the call to growproc():

uint64
sys_sbrk(void)
{
  int addr;
  int n;

  if(argint(0, &n) < 0)
    return -1;
  addr = myproc()->sz;
//   if(growproc(n) < 0) <- Remove the actual memory allocation here
//     return -1;
  myproc()->sz += n; // Still enlarge the current process's recorded size
  return addr;
}

Naturally, entering echo hi afterward produces a panic.

Lazy allocation

Implement lazy allocation for page tables. When a page fault occurs during trap handling, allocate a page for the faulting address.

The RISC-V manual and lab hints show that values 13 and 15 in the scause register represent page faults caused by attempted reads or writes:

In trap.c, inspect scause and perform additional handling when it is 13 or 15:

……
  } else if((which_dev = devintr()) != 0){
    // ok
  } else if((r_scause() == 13 || r_scause() == 15)){
    // do something here
  }
  else {
    printf("usertrap(): unexpected scause %p pid=%d\n", r_scause(), p->pid);
    printf("            sepc=%p stval=%p\n", r_sepc(), r_stval());
    p->killed = 1;
  }
……

The required handling allocates the missing user page. We can encapsulate it in a function named lazy_alloc().

Although a page fault reports an address, the entire page frame containing that address must be mapped to physical memory. Use PGROUNDDOWN first to find the beginning of that frame.

int lazy_alloc(uint64 va){
  struct proc *p = myproc();
  uint64 page_sta = PGROUNDDOWN(va);
  uint64* newmem = kalloc();
  if(newmem == 0){
    return -1;
  }
  memset(newmem, 0, PGSIZE);
  if(mappages(p->pagetable, page_sta, PGSIZE, (uint64)newmem, PTE_W|PTE_R|PTE_X|PTE_U) != 0){
    kfree(newmem);
    return -1;
  }
  
  return 0;
}

When calling mappages(), pay attention to the permissions. The page is accessible from user mode, so PTE_U must be set.

After these changes, running echo hi produces a panic in uvmunmap().

With lazy allocation, some pages may never be used before uvmunmap() attempts to remove them. Because such pages were never physically allocated, unmapping them causes a panic. We therefore need to modify uvmunmap():

void
uvmunmap(pagetable_t pagetable, uint64 va, uint64 npages, int do_free)
{
  uint64 a;
  pte_t *pte;

  if((va % PGSIZE) != 0)
    panic("uvmunmap: not aligned");

  for(a = va; a < va + npages*PGSIZE; a += PGSIZE){
    if((pte = walk(pagetable, a, 0)) == 0)
      continue; // Change panic to continue
      // panic("uvmunmap: walk");
    // uvmunmap is used while releasing a process, but this page may never have been allocated
    if((*pte & PTE_V) == 0)
      continue; // Change panic to continue
    //   panic("uvmunmap: not mapped");
    if(PTE_FLAGS(*pte) == PTE_V)
      panic("uvmunmap: not a leaf");
    if(do_free){
      uint64 pa = PTE2PA(*pte);
      kfree((void*)pa);
    }
    *pte = 0;
  }
}

This completes the basic part of the lab.

Lazytests and Usertests (moderate)

Make the lazy allocation implementation pass usertests and lazytests.

The lazy allocator just written still contains several bugs. This exercise asks us to fix them and pass both test suites.

The hints can be handled one at a time. First, support a negative argument to sbrk().

For a positive amount, we change only the process-size field and do not allocate actual space. For a negative amount, which reduces the process size, memory must truly be released so that other processes can use it. The implementation can be written as follows:

uint64
sys_sbrk(void)
{
  int addr;
  int n;
  struct proc *p = myproc();
  if(argint(0, &n) < 0)
    return -1;
  addr = p->sz;
  if(n < 0){
    if(p->sz + n < 0){ // A process cannot release more space than it owns
      return -1;
    }
    if(growproc(n) < 0){
      // growproc is actually called here to release space.
      printf("growproc err\n");
      return -1;
    }
  }else{
    myproc()->sz += n;
  }
  // if(growproc(n) < 0) 
  //   return -1;
  return addr;
}

The next hint is:

Kill a process if it page-faults on a virtual memory address higher than any allocated with sbrk().

That is, when the faulting address was never allocated through sbrk(), the kernel must not allocate a page there; it should kill the process instead.

We can write a helper that determines whether a virtual address belongs to a valid lazily allocated page:

int is_lazy_addr(uint64 va){
  struct proc *p = myproc();
  if(va < PGROUNDDOWN(p->trapframe->sp)
  && va >= PGROUNDDOWN(p->trapframe->sp) - PGSIZE
  ){
    // Exclude the guard page, which is discussed later
    return 0;
  }
  if(va > MAXVA){
    return 0;
  }
  pte_t* pte = walk(p->pagetable, va, 0);
  
  if(pte && (*pte & PTE_V)){
    return 0;
  }  

  if(va >= p->sz){
    return 0;
  }

  return 1;
}

First, a page with PTE_V set is clearly not lazily allocated because it already has a valid mapping.

Next, if va >= p->sz, the address was never requested through sbrk(), so it is not a valid lazy-allocation address.

Adding this helper to the condition in trap.c gives:

……
  } else if((which_dev = devintr()) != 0){
    // ok
  } else if((r_scause() == 13 || r_scause() == 15) && is_lazy_addr(r_stval())){ // Add is_lazy_addr here
    // Allocate memory directly for a page fault
    uint64 fault_addr = r_stval();
      if(lazy_alloc(fault_addr) < 0){
        p->killed = 1;
      }
  }
  else {
    printf("usertrap(): unexpected scause %p pid=%d\n", r_scause(), p->pid);
    printf("            sepc=%p stval=%p\n", r_sepc(), r_stval());
    p->killed = 1;
  }
……

The next requirement is:

Handle the parent-to-child memory copy in fork() correctly.

This means that memory copying from the parent into the child during fork() must work with lazily allocated pages.

Reading fork() shows that uvmcopy() in vm.c performs this copy. It fails under lazy allocation because some parent page frames have never actually been allocated, and trying to copy them causes a panic. As with uvmunmap(), skip such lazy pages by replacing the panics with continue:

int
uvmcopy(pagetable_t old, pagetable_t new, uint64 sz)
{
  pte_t *pte;
  uint64 pa, i;
  uint flags;
  char *mem;

  for(i = 0; i < sz; i += PGSIZE){
    if((pte = walk(old, i, 0)) == 0)
      continue;   // Change panic to continue here.
      // panic("uvmcopy: pte should exist");
    if((*pte & PTE_V) == 0)
      continue;
      // panic("uvmcopy: page not present");
    pa = PTE2PA(*pte);
    flags = PTE_FLAGS(*pte);
    if((mem = kalloc()) == 0)
      goto err;
    memmove(mem, (char*)pa, PGSIZE);
    if(mappages(new, i, PGSIZE, (uint64)mem, flags) != 0){
      kfree(mem);
      goto err;
    }
  }
  return 0;

 err:
  uvmunmap(new, 0, i / PGSIZE, 1);
  return -1;
}

The next hint says:

Handle the case in which a process passes a valid address from sbrk() to a system call such as read or write, but the memory for that address has not yet been allocated.

This hint was honestly hard to understand, and I searched online for a long time. Some system calls write data to a user virtual address, such as write(). If that address is lazy, the access causes a page fault. A user-mode page fault is fine because our handler deals with it. A kernel-mode exception, however, causes an immediate panic, as explained in the xv6 notes.

System calls use copyin() and copyout() to read or write user virtual addresses. Examine one of these functions:

// Copy from user to kernel.
// Copy len bytes to dst from virtual address srcva in a given page table.
// Return 0 on success, -1 on error.
int
copyin(pagetable_t pagetable, char *dst, uint64 srcva, uint64 len)
{
  uint64 n, va0, pa0;

  while(len > 0){
    va0 = PGROUNDDOWN(srcva);
    pa0 = walkaddr(pagetable, va0); // Notice this line
    if(pa0 == 0)
      return -1;
    n = PGSIZE - (srcva - va0);
    if(n > len)
      n = len;
    memmove(dst, (void *)(pa0 + (srcva - va0)), n);

    len -= n;
    dst += n;
    srcva = va0 + PGSIZE;
  }
  return 0;
}

Both functions call walkaddr() to find the physical address corresponding to a user virtual address. walkaddr() is implemented as:

// Look up a virtual address, return the physical address,
// or 0 if not mapped.
// Can only be used to look up user pages.
uint64
walkaddr(pagetable_t pagetable, uint64 va)
{
  pte_t *pte;
  uint64 pa;

  if(va >= MAXVA)
    return 0;

  pte = walk(pagetable, va, 0);
  
  if(pte == 0)
    return 0;
  if((*pte & PTE_V) == 0)
    return 0;
  if((*pte & PTE_U) == 0)
    return 0;
  pa = PTE2PA(*pte);
  return pa;
}

walkaddr() calls walk() and returns zero immediately when no mapping is found.

The behavior also makes sense from the function’s purpose. A lazily allocated page frame does not yet have any physical address, so looking up that address naturally returns zero.

If va belongs to a lazy page, walk() necessarily returns zero. The following implementation shows why:

pte_t *
walk(pagetable_t pagetable, uint64 va, int alloc)
{
  if(va >= MAXVA)
    panic("walk");

  for(int level = 2; level > 0; level--) {
    pte_t *pte = &pagetable[PX(level, va)];
    if(*pte & PTE_V) { // Check whether the address has been allocated.
                       // If not and alloc is zero, return zero.
      pagetable = (pagetable_t)PTE2PA(*pte);
    } else {
      if(!alloc || (pagetable = (pde_t*)kalloc()) == 0)
        return 0;
      memset(pagetable, 0, PGSIZE);
      *pte = PA2PTE(pagetable) | PTE_V;
    }
  }
  return &pagetable[PX(0, va)];
}

We can modify walkaddr() to check whether the current va belongs to a lazy page. If it does, allocate a physical page before returning zero and then continue the normal lookup. Once physical memory has been allocated, its address can be found.

// Look up a virtual address, return the physical address,
// or 0 if not mapped.
// Can only be used to look up user pages.
uint64
walkaddr(pagetable_t pagetable, uint64 va)
{
  pte_t *pte;
  uint64 pa;

  if(va >= MAXVA)
    return 0;
  
  if(is_lazy_addr(va)){ // If this is lazy allocation, allocate physical memory first.
    lazy_alloc(va);
  }
  pte = walk(pagetable, va, 0);
  
  if(pte == 0)
    return 0;
  if((*pte & PTE_V) == 0)
    return 0;
  if((*pte & PTE_U) == 0)
    return 0;
  pa = PTE2PA(*pte);
  return pa;
}

The fifth hint is:

Handle out-of-memory correctly: if kalloc() fails in the page fault handler, kill the current process.

In other words, if no physical page is available, kill the current process.

This is already implemented in trap.c:

uint64 fault_addr = r_stval();
if(lazy_alloc(fault_addr) < 0){
  p->killed = 1;
}

If lazy_alloc() fails because memory is exhausted, the process is killed.

The final hint is:

Handle faults on the invalid page below the user stack.

This requires reviewing the page-table chapter. The following diagram shows the user-mode memory layout:

Immediately below the stack is a guard page whose PTE_V bit is not set. Accessing it from user mode triggers a page fault. That mechanism worked before, but lazy allocation now responds to a fault by allocating physical memory rather than killing the process.

The guard page exists to prevent memory overflow and must not receive a physical page. Add a test to is_lazy_addr(): if an address belongs to the guard page, it is not a valid lazy-allocation address.

if(va < PGROUNDDOWN(p->trapframe->sp)            // Use the user stack pointer sp to locate the stack's virtual address.
                                                 // The guard page is immediately below the stack, so use
                                                 // PGROUNDDOWN(p->trapframe->sp) as its upper boundary.
&& va >= PGROUNDDOWN(p->trapframe->sp) - PGSIZE
){
  return 0;
}

After this change, the tests pass. I wish everyone working on this lab an early AC:

Summary

I need to improve my debugging ability. This lab genuinely took me a very long time to debug…

[MIT 6.s081] Xv6 Lab 4: Traps Record

2022-07-25T00:00:00.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Preface: today is 2022/7/25, so first I want to celebrate the blog having operated for 100 days.

Update on 2022/9/14: I recently put the lab code on GitHub. If you need a reference, you can find it here:

https://github.com/ttzytt/xv6-riscv

The different branches contain the different labs.

Lab 4: traps

RISC-V assembly

~~Postponed for now.~~

Backtrace

Implement a backtrace() function. When a program calls it, the function should print that program’s “function call order”—that is, all function addresses currently on the stack, in sequence.

The most important prerequisite for this exercise is understanding the process of a function call. For details, refer to this article that I wrote earlier.

I have placed the most important diagram and video from that article below. This is ~~definitely not padding the word count~~. If you were previously familiar with function calls but have forgotten the details, these should make them easy to recall.

The lab asks us to print a chain of function calls.

For example, consider this program:

int third(int x){
    backtrace();
    return x;
}
int second(int x){
    return third(x); // Assume the address is 114
}
int first(int x){ 
    return second(x); // Assume the address is 514
} 

int main(){
    int test = first(114514); // Assume the address is 1919
}

The correct output from backtrace() should be:

1
2
3

114
514
1919

In other words, recursively print the addresses of the calling functions.

Every stack frame stores the current function’s return address—the location to which execution should return after that function finishes.

We can therefore print the return address from each frame. We also need a variable storing the current frame pointer. Adding the appropriate offset to this pointer obtains the preceding function’s frame pointer, allowing us to print its return address as well.

One detail is that my earlier article used an x86-64 processor, where the frame pointer is named the bp, or base pointer, register. In RISC-V, the fp, or frame pointer, register performs the same job.

The location to which fp points in RISC-V is also slightly different from the x86 location. The following diagram shows the layout.^[1]

High addresses

Stack
                   .
                   .
      +->          .
      |   +-----------------+   |
      |   | return address  |   |
      |   |   previous fp ------+
      |   | saved registers |
      |   | local variables |
      |   |       ...       | <-+
      |   +-----------------+   |
      |   | return address  |   |
      +------ previous fp   |   |
          | saved registers |   |
          | local variables |   |
      +-> |       ...       |   |
      |   +-----------------+   |
      |   | return address  |   |
      |   |   previous fp ------+
      |   | saved registers |
      |   | local variables |
      |   |       ...       | <-+
      |   +-----------------+   |
      |   | return address  |   |
      +------ previous fp   |   |
          | saved registers |   |
          | local variables |   |
  $fp --> |       ...       |   | <-- Notice this!!!
          +-----------------+   |
          | return address  |   |  
          |   previous fp ------+ <-- On x86, the bp pointer would point here
          | saved registers |
  $sp --> | local variables |
          +-----------------+

Low addresses (growth direction)

In RISC-V, fp points to a location immediately above the current frame’s return address, meaning a higher address. In x86, bp points to the saved bp of the preceding stack frame.

This probably results from a difference in how x86 and RISC-V define a stack frame. In RISC-V’s definition, the return address is part of the current stack frame, which honestly seems like the more reasonable design to me.

Although fp always lets us find a function’s return address, we still need to obtain the current value of fp. This requires inline assembly in C. We can put the following helper in kernel/riscv.h:

static inline uint64
r_fp()
{
  uint64 x;
  asm volatile("mv %0, s0" : "=r" (x) );
  return x;
}

The basic format of GCC extended inline assembly is:

asm asm-qualifiers ( AssemblerTemplate 
                 : OutputOperands 
                 [ : InputOperands
                 [ : Clobbers ] ])

Here, asm begins the inline assembly and asm-qualifiers describes its properties. The volatile qualifier used above tells GCC not to optimize this assembly away.

In ("mv %0, s0" : "=r" (x)), mv %0, s0 is an assembly template rather than final assembly, somewhat like a C++ template. During compilation, GCC replaces %0 with the register that contains the variable selected by the later constraint, : "=r" (x), which is x here.

The string "=r" is a constraint. The r says that x may reside in any general-purpose register, while the equals sign says that this operand is written by the assembly.

Many constraints besides r exist.^[2] For example, m allows the variable to reside in memory. For additional constraints, see the GCC documentation.

The GCC extended-assembly documentation also explains this feature in considerable detail.

Overall, r_fp() reads register s0, stores the value in x, and returns x.

But we want the fp register, so why does the function use s0? The following table provides the answer.^[3]

The ABI Name column shows that s0 is an alias for fp.

With this knowledge, we can write backtrace():

void 
backtrace(){
  printf("in bt\n");
  // Below the frame pointer is the return address.
  // Below that is the preceding stack frame's frame pointer.
  uint64* cur_frame = (uint64 *)r_fp();
  uint64* top = PGROUNDUP((uint64)cur_frame);
  uint64* bot = PGROUNDDOWN((uint64)cur_frame);
  while(cur_frame < top && cur_frame > bot){
    printf("%p\n", cur_frame[-1]); // First print the current return address
    cur_frame = cur_frame[-2]; // Then move from the current frame to the preceding frame
  }
}

Some expressions here look unusual, almost like negative array indices. In fact, cur_frame[-1] is equivalent to *(cur_frame - 1). Because cur_frame is a pointer to 64-bit values, this reads the data eight bytes before cur_frame.

PGROUNDDOWN and PGROUNDUP are used because a chain of function calls fits within at most one page. If recursive printing goes outside that page’s range, we have reached the bottommost function and can stop.

Finally, add backtrace() to the sys_sleep() system call as requested, and this part is complete.

Alarm

Implement a sigalarm(interval, handler) system call that executes handler once every interval clock ticks. Also implement sigreturn(): when the handler calls it, execution of the handler should stop and the normal instruction sequence should resume. Passing zero for both arguments of sigalarm disables the handler.

This lab is rather difficult to understand, especially sigreturn. Examine alarmtest.c carefully. A good understanding of the trap process is also necessary; if unfamiliar, see this article.

void
periodic()
{
  count = count + 1;
  printf("alarm!\n");
  sigreturn();
}

// tests whether the kernel calls
// the alarm handler even a single time.
void
test0()
{
  int i;
  printf("test0 start\n");
  count = 0;
  sigalarm(2, periodic);
  for(i = 0; i < 1000*500000; i++){
    if((i % 1000000) == 0)
      write(2, ".", 1);
    if(count > 0)
      break;
  }
  sigalarm(0, 0);
  if(count > 0){
    printf("test0 passed\n");
  } else {
    printf("\ntest0 failed: the kernel never called the alarm handler\n");
  }
}

sigreturn means that execution may originally be inside this for loop and then abruptly begin executing periodic() because the interval elapsed. If periodic() calls sigreturn(), execution inside periodic() should stop and resume in the for loop. This video creator explains it clearly.

We can examine the tests in alarmtest.c in order and implement the calls according to their requirements.

test0: invoke handler

Get started by modifying the kernel to jump to the alarm handler in user space, which will cause test0 to print “alarm!”. Don’t worry yet what happens after the “alarm!” output; it’s OK for now if your program crashes after printing “alarm!”. Here are some hints:

In other words, first jump correctly into user mode to execute the handler. To preserve isolation, the function cannot simply run in the kernel. A crash after the jump is acceptable for now.

Recall the xv6 trap process. The epc register determines the address to which execution returns after a trap. Changing epc directly makes the return jump to the handler’s address.

How do we determine when the interval has elapsed?

RISC-V hardware—I am not entirely sure which hardware component—generates a timer interrupt every clock tick, and trap.c handles it.

We can count these interrupts to decide whether to jump. When required, directly replace epc in the trapframe in trap.c with the handler’s address.

Add the following fields to struct proc for every process:

uint64 alarm_tks; stores the handler interval; zero means disabled.
void (*alarm_handler)(); stores the handler address.
uint64 alarm_tk_elapsed; stores the time elapsed since the handler last ran.

sys_sigalarm() stores its arguments in these fields. For now, sys_sigreturn() does nothing and simply returns zero:

uint64 
sys_sigalarm(void){
  int ticks;
  struct proc* p = myproc();
  uint64 handler;
  try(argint(0, &ticks), return -1);
  try(argaddr(1, &handler), return -1);
  p->alarm_tks = ticks;
  p->alarm_handler = handler;
  p->alarm_tk_elapsed = 0;
  return 0;
}

Because these fields have been added, initialize and release them appropriately in the process initialization function allocproc() and cleanup function freeproc().

First, the change in allocproc():

……
  p->alarm_tk_elapsed = 0;
  p->alarm_state = 0;
  p->alarm_tks = 0;
  return p;
}

Then freeproc():

……
  p->alarm_handler = 0;
  p->alarm_tk_elapsed = 0;
  p->alarm_tks = 0;
}

We can now implement the jump in usertrap() in trap.c:

……
  if(which_dev == 2){ // The timer interrupt number is 2
    if(p->alarm_tks > 0){ 
      p->alarm_tk_elapsed++; // Time elapsed since the handler last ran
      if(p->alarm_tk_elapsed > p->alarm_tks){ // The specified interval has elapsed
        p->alarm_tk_elapsed = 0;
        p->trapframe->epc = p->alarm_handler; // Replace epc so user mode executes the instruction at that address
      }
    }
    yield();
  }

This successfully jumps to the handler and passes test0, although it predictably crashes afterward.

The main reason for the crash is that sys_sigreturn() has not yet been implemented, so after the handler finishes the process does not know where to return.

Passing test1 and test2 requires solving this problem.

test1/test2(): resume interrupted code

Chances are that alarmtest crashes in test0 or test1 after it prints “alarm!”, or that alarmtest (eventually) prints “test1 failed”, or that alarmtest exits without printing “test1 passed”. To fix this, you must ensure that, when the alarm handler is done, control returns to the instruction at which the user program was originally interrupted by the timer interrupt. You must ensure that the register contents are restored to the values they held at the time of the interrupt, so that the user program can continue undisturbed after the alarm. Finally, you should “re-arm” the alarm counter after each time it goes off, so that the handler is called periodically.

In short, execution must return to the correct location after the handler finishes.

Register values change while entering the kernel to handle traps and system calls. Merely restoring epc to the correct address is therefore insufficient: the complete register environment also needs to be backed up.

Add another struct trapframe field to struct proc to preserve the environment from before the handler:

……
struct trapframe *trapframe; // data page for trampoline.S
struct trapframe *alarmframe; // Newly added backup trapframe
……

Naturally, it must also be allocated and released in allocproc() and freeproc().

In allocproc():

……
if((p->alarmframe = (struct trapframe *)kalloc()) == 0){
  freeproc(p);
  release(&p->lock);
  return 0;
}
……

In freeproc():

1
2
3

if(p->alarmframe)
  kfree((void*)p->alarmframe);
p->alarmframe = 0;

usertrap() in trap.c can populate alarmframe. When the handler must run, back up the environment first and then redirect execution:

if(which_dev == 2){
  if(p->alarm_tks > 0){
    p->alarm_tk_elapsed++;
    if(p->alarm_tk_elapsed > p->alarm_tks){
      p->alarm_tk_elapsed = 0;
      *p->alarmframe = *p->trapframe; // Notice this line
      p->trapframe->epc = p->alarm_handler;
    }
  }
  yield();
}

sys_sigreturn() restores the trapframe from alarmframe. This restores epc and all general-purpose registers, naturally leaving the handler and resuming the program’s original sequence:

uint64
sys_sigreturn(void){
  struct proc* p = myproc();
  *p->trapframe = *p->alarmframe;
  return 0;
}

Running alarmtest again still does not pass every test.

Imagine that a handler executes extremely slowly. The specified number of ticks may pass again before the previous handler invocation finishes. If we replace epc again at that point, the handler restarts from its beginning. This is disastrous: epc keeps being reset and the handler can never finish.

The test program includes exactly this case:

void
slow_handler()
{
  count++;
  printf("alarm!\n");
  if (count > 1) {
    printf("test2 failed: alarm handler called more than once\n");
    exit(1);
  }
  for (int i = 0; i < 1000*500000; i++) { // An extremely slow handler
    asm volatile("nop"); // avoid compiler optimizing away loop
  }
  sigalarm(0, 0);
  sigreturn();
}

We therefore need another field in struct proc, named alarm_state. A value of one means that the handler is currently executing. Even if another interval elapses, epc must not be changed to run the handler again while this state is active.

Because a new field has been added, allocproc and freeproc must be updated as well; I will not repeat those straightforward changes.

The more important change is in usertrap():

if(which_dev == 2){
  if(p->alarm_tks > 0){
    p->alarm_tk_elapsed++;
    if(p->alarm_tk_elapsed > p->alarm_tks && !p->alarm_state){ // alarm_state must be zero here
      p->alarm_tk_elapsed = 0;
      *p->alarmframe = *p->trapframe;
      p->trapframe->epc = p->alarm_handler;
      p->alarm_state = 1; // Changing epc means execution has begun
    }
  }
  
  yield();
}

sys_sigreturn() also needs a change because calling it means that the handler is no longer running:

uint64
sys_sigreturn(void){
  struct proc* p = myproc();
  *p->trapframe = *p->alarmframe;
  p->alarm_state = 0; // Set alarm_state to zero because the handler has stopped
  return 0;
}

After these changes, the lab passes. I also wish everyone working on it an early AC:

Summary

More important than the exercises themselves is understanding the trap process in xv6. Even without completely understanding it, one can follow the instructions step by step and finish the lab. Understanding the mechanism is genuinely difficult, however, because it involves a great deal of unfamiliar RISC-V assembly and low-level knowledge. Once understood and implemented, it is hard not to admire the ingenuity of operating-system design.

Completing this lab also resolved many of my earlier questions about operating systems, including the principle behind the alarm exercise. At the same time, it revealed how shallow my understanding of assembly still is. See the xv6 notes in this article. I could never understand why userret and uservec exchange the sscratch register. After asking someone, I learned that it is a privileged register and cannot be manipulated using instructions such as ld and sd, although I still do not understand the reason for that design.

CC (CodeChef) STARTERS 48 Solutions

2022-07-20T21:58:10.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Accurate XOR

Approach

Problem link

This problem uses a property of XOR. When XORing multiple consecutive 0s or 1s, only an odd number of 1s makes the result 1.

If there are an even number of 1s, every 1 can always be paired with another 1 so that their XOR becomes 0. The occurrence of 0 does not affect the final result, so if there are an even number of 1s, the final result is always 0.

The Xor-value of a node is defined as the bitwise XOR of all the binary values present in the subtree of that node.

This sentence in the statement says that the XOR value of a tree is the XOR sum of every node under that tree.

In other words, let the current tree’s root be $r$ , and let $r$ have $x$ child nodes (including indirect children, such as children in its subtrees), whose values are $c_1 \sim c_x$ . Then the XOR value of $r$ is:

$\operatorname{XOR}(r) = c_1 \oplus c_2 \ldots \oplus c_{x - 1} \oplus c_x$

Because every child node has a value of either 1 or 0, the property above tells us that if the current tree’s XOR value is 1, then an odd number of nodes in all its subtrees have value 1, and vice versa.

That is, if the XOR value of tree $r$ is 1:

$\sum_{x=1}^{n}c_x \bmod 2 = 1$

The problem requires $k$ subtrees to have an XOR value of 1. Therefore, for every child node in these $k$ subtrees, the sum of their values must be odd.

Let $\text{odcnt}_i$ be the number of child nodes with value 1 in tree $i$ . Let the current tree be $r$ , and suppose we still need $kl$ trees to have an XOR value of 1 (that is, some trees already have an XOR value of 1).

If $kl > 0$ and $\text{odcnt}_r \bmod 2 = 0$ , meaning that an even number of its child nodes have value 1, we should set the value of this node to 1.

This is because $kl > 0$ , so we need more trees with XOR value 1. Since this tree has an even number of child nodes with value 1, its XOR value is not 1. Changing the value of the tree itself to 1 changes its XOR value to 1, achieving our goal.

Conversely, if $kl = 0$ , we do not need more trees with XOR value 1. However, if $\text{odcnt}_r \bmod 2 = 1$ , meaning the sum of the values of all its child nodes is odd, we should set $r$ to 1.

This is because we do not want to produce more trees with XOR value 1. Setting $r$ to 1 makes the sum of the values of all its nodes even, and the XOR value of $r$ becomes $0$ .

With these two conclusions, we can use DFS to find the answer.

Code

// tzyt
#include 
using namespace std;
#define ll long long
const int MAXN = 2e5 + 10;
vector<int> e[MAXN];
// k subtrees of odd size.
int od_cnt[MAXN];
int n, k;

void dfs(int cur, string& ans) {
    for (int nex : e[cur]) {
        dfs(nex, ans);
        od_cnt[cur] += od_cnt[nex];
    }
    if (k) {
        if ((od_cnt[cur] & 1) == 0) {  // An even number of nodes in the subtree have value 1.
            // Change it to an odd number.
            ans[cur] = '1';
            od_cnt[cur]++;
        }
        k--;
    } else { // The condition is already satisfied, but there may be one extra.
        if(od_cnt[cur] & 1){ // An odd number of child nodes have value 1.
            ans[cur] = '1';
            od_cnt[cur]++;
        }
    }
}

int main() {
    int t;
    cin >> t;
    while (t--) {
        
        cin >> n >> k;
        for_each(e + 1, e + 1 + n, [](vector<int>& a) { a.clear(); });
        string ans;
        ans.resize(n + 1);
        for_each(ans.begin(), ans.end(), [](char &a){a = '0';});
        fill(od_cnt + 1, od_cnt + 1 + n, 0); // Reset data.

        for (int i = 2; i <= n; i++) {
            int tmp;
            cin >> tmp;
            e[tmp].push_back(i);
        }

        dfs(1, ans); 
        for (int i = 1; i <= n; i++) {
            cout << ans[i];
        }
        cout << '\n';
    }
}

Strict Permutation

Approach

Problem link

My original idea was to sort every constraint by position, and then by value if the positions were equal.

Then I would traverse every constraint and alternately insert each constraint and each unrestricted value (according to their values, because the problem asks for the lexicographically smallest result). The explanation here is probably unclear; the following was my previous code:

/*Date: 22 - 07-20 20 10
PROBLEM_NUM: */
#define FDEBUG
#if (defined FDEBUG) && (!defined ONLINE_JUDGE)
#define DEBUG(fmt, ...) fprintf(stderr, fmt, ##__VA_ARGS__)
#define DWHILE(cnd, blk) \
    while (cnd) blk
#define DFOR(ini, cnd, itr, blk) \
    for (ini; cnd; itr) blk
#else
#define DEBUG(fmt, ...)
#define DWHILE(cnd, blk)
#define DFOR(ini, cnd, itr, blk)
#endif

#include 
using namespace std;
#define ll long long
#define pause system("pause")
#define IINF 0x3f3f3f3f
#define rg register
// keywords:

struct Constrain {
    int val, pos;
    bool operator<(Constrain b) const {
        if (pos != b.pos) return pos < b.pos;
        return val < b.val;
    }
    bool operator>(Constrain b) const { return b < *this; }
};

int main() {
    int t;
    cin >> t;
    while (t--) {
        int n, m;
        cin >> n >> m;
        priority_queue, greater> pq;
        vector<int> ans;
        ans.reserve(n);
        set<int> ncons;
        for (int i = 1; i <= n; i++) {
            ncons.insert(i);
        }
        for (int i = 0; i < m; i++) {
            Constrain tmp;
            cin >> tmp.val >> tmp.pos;
            pq.push(tmp);
            ncons.erase(tmp.val);
        }
        while (pq.size()) {
            auto tp = pq.top();
            pq.pop();
            bool used = false;
            if (ans.size() >= tp.pos) {
                goto FAIL;
            }
            
            while (ans.size() < tp.pos - 1) {
                int ist = *ncons.begin();
                if (tp.val < ist) {
                    ans.push_back(tp.val);
                    used = true;
                } else {
                    ans.push_back(ist);
                    ncons.erase(ist);
                }
            }
            if (!used) {
                ans.push_back(tp.val);
            }
        }
        while (ncons.size()) {
            int ist = *ncons.begin();
            ans.push_back(ist);
            ncons.erase(ist);
        }

    SUCC:
        for (int cur : ans) {
            cout << cur << ' ';
        }
        cout << '\n';
        continue;
    FAIL:
        cout << "-1\n";
    }
    pause;
}

This reckless approach causes a problem. Suppose we sort the constraints as described above and call them $c_{1 \sim m}$ .

Then the number in $c_i$ can only appear in the interval $(c_{i - 1}, c_i]$ , which does not satisfy the problem’s requirements. This is why it received so many WAs.

The correct solution is to calculate from back to front.

We maintain a max-heap $pq$ and traverse every position (the positions in the permutation) from back to front.

If the position of some constraint is the current position, we add the value of that constraint to $pq$ . For every position traversed, we can then directly take the top element from $pq$ and put it into the answer.

Therefore, we can take the value of a constraint from $pq$ only when the current position is smaller than the constraint’s position, so every element taken from $pq$ is valid.

At the same time, these elements are also the largest possible. Since we traverse from back to front, this ensures that the resulting permutation is lexicographically smallest.

Finally, we need to consider when to output -1. Since $pq$ stores every element that is valid for the current position, if nothing can be taken from $pq$ , then no valid permutation can be produced.

One final point: for numbers without any constraints, we can add them to $pq$ at the beginning, or equivalently, their constraint position is $n$ .

Code

// tzyt
#include 
using namespace std;
// keywords:
int main() {
    int t;
    cin >> t;
    while (t--) {
        int n, m;
        cin >> n >> m;
        vector<int> lim(n + 1, n), ans(n + 1);  
        // By default, only the position before n is required (there is no constraint).
        vectorint>> lislim(n + 1);
        // lislim[i] stores all values whose constraint position is i.
        for (int i = 1; i <= m; i++) {
            int val, pos;
            cin >> val >> pos;
            lim[val] = pos;
        }
        for (int i = 1; i <= n; i++) {
            lislim[lim[i]].push_back(i);
        }
        priority_queue<int> pq;
        for (int i = n; i >= 1; i--) {
            for (int cur : lislim[i]) {
                // Reaching a constraint point makes new numbers available.
                pq.push(cur);
            }
            if (pq.empty()) { // Empty means there is no valid element.
                goto FAIL;
            }
            ans[i] = pq.top();
            pq.pop();
        }
    SUCC:
        for (int i = 1; i <= n; i++) {
            cout << ans[i] << ' ';
        }
        cout << '\n';
        continue;
    FAIL:
        cout << "-1\n";
    }
}

CF1705 B, C, D1 Solutions

2022-07-18T21:53:34.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

B. Making Towers

Approach

Observe the figure provided for the first sample in the problem statement:

We can see that, if we want blocks of one color to form a tower, unless multiple blocks of the same color are adjacent in array $c$ and can be placed directly upward, we must place some blocks of other colors to both sides after placing a block of that color, then place blocks in the opposite direction, and finally make the two blocks of the same color lie on a straight line. It looks approximately like this:

⬆->->->A
⬆<-<-<-A<-<-<-⬆
        A->->->⬆
        1 2 ... z

Here, $A$ represents a tower of one color, while the arrows represent the path along which colored blocks are placed.

Observation shows that an even number of blocks of other colors must be placed between two $A$ s. The explanation follows:

Suppose the first $A$ is at $(x, y)$ , and the number of blocks of other colors that we place to the right (it can also be to the left) is $z$ .

Then, to put the second $A$ at $(x, y + 1)$ , colored blocks need to be placed at $(x + 1, y) \sim (x + z, y)$ and $(x + 1, y + 1) \sim (x + z, y + 1)$ . There are $2z$ blocks in total, so the number is even (if blocks are stacked directly upward, it is $0$ , which is also even).

This means that, suppose there are two blocks of the same color, $A$ and $B$ , at positions $i$ and $j$ in array $c$ . Only when $\lvert i - j \rvert$ is odd can $A$ be stacked on top of $B$ , or $B$ on top of $A$ .

Moreover, $\lvert i - j \rvert$ can only be odd when $i$ and $j$ have different parity.

We can then solve the problem using DP. We repeat the same DP process for every color (in fact, it is more like a recurrence). Let $dp_i$ be the greatest height of a tower that can be built using $i$ blocks of this color in array $c$ .

Then $dp_i$ can transition from $dp_{0 \sim i - 1}$ (with $+1$ ), and, as described above, $dp_i$ and $dp_{0 \sim i - 1}$ should have different parity.

At the same time, we need to find the nearest block with different parity; otherwise, blocks may be wasted, or another block may be placed at a position that was already used earlier.

Code

// author: tzyt
#include 
using namespace std;
#define ll long long
int main() {
    int t;
    cin >> t;
    while (t--) {
        int n;
        cin >> n;
        int c[n + 1];
        vector<int> cpos[n + 1], ans(n + 1);
        set<int> unqc; // Store all distinct colors.
        for (int i = 1; i <= n; i++) {
            cin >> c[i];
            cpos[c[i]].push_back(i);
            unqc.insert(c[i]);
        }
        int dp[n + 1];
        for (int cur : unqc) {
            fill(dp, dp + cpos[cur].size(), 1); 
            // No matter what, as long as there is a block, a tower of height 1 can always be built.
            int mx = 1;
            // The maximum value in dp[0 ~ cpos[cur].size()].
            int lstod = -1, lstev = -1;
            // The nearest odd and even positions; -1 is the initial value.
            cpos[cur][0] & 1 ? lstod = 0 : lstev = 0;
            // Determine the parity of the first one.
            for (int i = 1; i < cpos[cur].size(); i++) {
                int lst = cpos[cur][i] & 1 ? lstev : lstod;
                if (lst != -1) 
                    dp[i] = dp[lst] + 1;
                // lst is the first position with different parity.
                mx = max(dp[i], mx);
                cpos[cur][i] & 1 ? lstod = i : lstev = i;
                // Update the nearest odd and even positions.
            }
            ans[cur] = mx;
        }

        for (int i = 1; i <= n; i++) {
            cout << ans[i] << ' ';
        }
        cout << '\n';
    }
}

C. Qpwoeirut And The City

Approach

We can see that, no matter what, there can be at most $\lfloor \frac{n - 1}{2} \rfloor$ cool houses in the city.

If the number of houses is odd, only one arrangement can achieve this many cool houses. It is the arrangement shown by the first sample.

Beginning with the second house, make every house at an even position cool; that is, cool and non-cool houses appear alternately.

The cost of turning an ordinary house into a cool house can be calculated as follows:

inline ll calc_cost(int i, int* h) {
    if (h[i] <= h[i - 1] || h[i] <= h[i + 1])
        return max(h[i - 1], h[i + 1]) - h[i] + 1;
    else
        return 0;
}

That is, make the current house one unit higher than the taller of its adjacent houses.

However, the case of an even number of houses is more complicated. In this case, $\lfloor \frac{n - 1}{2} \rfloor$ must equal $\frac{n}{2} - 1$ .

Then there are $\frac{n}{2} + 1$ non-cool houses, so two adjacent non-cool houses must appear. These two consecutive non-cool houses can appear at any position, and we need to consider all cases.

For example, if $n = 8$ , there are the following arrangements:

$\texttt{010101}\textcolor{red}{\texttt{00}}\\\texttt{0101}\textcolor{red}{\texttt{00}}\texttt{10}\\\texttt{01}\textcolor{red}{\texttt{00}}\texttt{1010}\\\textcolor{red}{\texttt{00}}\texttt{101010}$

However, calculating every case from beginning to end would take too much time.

Therefore, we can calculate only the change in cost from one case to another.

For example:

$\texttt{010101}\textcolor{red}{\texttt{00}}\\\downarrow\\\texttt{0101}\textcolor{red}{\texttt{00}}\texttt{10}$

During this process, the sixth house changes from cool to non-cool, while the seventh house changes from non-cool to cool.

Suppose we are currently changing house $i$ from cool to non-cool and house $i + 1$ from non-cool to cool. We only need to call the preceding calc_cost, subtract the cost of $i$ , and then add the cost of $i + 1$ .

// author: tzyt
#include 
using namespace std;
#define ll long long

inline ll calc_cost(int i, int* h) {
    if (h[i] <= h[i - 1] || h[i] <= h[i + 1])
        return max(h[i - 1], h[i + 1]) - h[i] + 1;
    else
        return 0;
}

int main() {
    int t;
    cin >> t;
    while (t--) {
        int n;
        cin >> n;
        int h[n + 1];
        for (int i = 1; i <= n; i++) {
            cin >> h[i];
        }
        ll ans = 0, tmp = 0;
        
        // Solution for the odd case.
        for (int i = 2; i < n; i += 2) {
            ans += calc_cost(i, h);
        }
        if (n & 1) {
            cout << ans << '\n';
            continue;
        }

        tmp = ans;
        for (int i = n - 2; i >= 2; i -= 2) {
            // Enumerate the position of the consecutive 0s.
            tmp -= calc_cost(i, h);
            tmp += calc_cost(i + 1, h);
            ans = min(ans, tmp);
        }
        cout << ans << '\n';
    }
}

D1. Chopping Carrots (Easy Version)

Approach

Let us try setting the minimum $\lfloor \frac{a_i}{p_i} \rfloor$ to $mn$ . Then $mn \in [0, a_1]$ , because the minimum value of $p_i$ is $1$ , so $\lfloor \frac{a_1}{1}\rfloor$ is $a_1$ .

On this basis, we greedily try to make every $\lfloor \frac{a_i}{p_i} \rfloor$ as close to $mn$ as possible. This makes the maximum $\lfloor \frac{a_i}{p_i} \rfloor$ as small as possible.

In this way, we can calculate $p_i$ . Because $\lfloor \frac{a_i}{p_i} \rfloor \ge mn$ , we have $p_i = \lfloor \frac{a_i}{mn} \rfloor$ . Of course, $p_i$ cannot be greater than $k$ , and if $mn = 0$ , we let $p_i = k$ .

We then enumerate every possible $mn$ and calculate the maximum $\lfloor \frac{a_i}{p_i} \rfloor$ in that case to obtain the answer. The approach seems quite concise, but it really is difficult to think of.

Code

// author: tzyt
#include 
using namespace std;
#define IINF 0x3f3f3f3f
int main() {
    int t;
    cin >> t;
    while (t--) {
        int n, k;
        cin >> n >> k;
        int a[n + 1];
        for (int i = 1; i <= n; i++) {
            cin >> a[i];
        }
        int ans = IINF;
        int mxv = 0;
        for (int mnv = 0; mnv <= a[1]; mnv++) {
            // Enumerate mn.
            for (int i = 1; i <= n; i++) {
                int p = min(k, (mnv ? (a[i] / mnv) : k));
                // mnv ? (a[i] / mnv) : k handles the case where mnv is 0.
                mxv = max(mxv, a[i] / p);
            }
            ans = min(ans, mxv - mnv);
        }
        cout << ans << '\n';
    }
}

CF1705 C, D Solutions

2022-07-15T16:59:16.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

C. Mark and His Unfinished Essay

Approach

Given these constraints, we obviously cannot actually copy the string, so we need to find another method.

We can see that every segment newly appended to the end of the string has an identical counterpart earlier in the string, located by an offset.

For example, consider the final insertion in the first sample:

$\texttt{mark\ mark\ mar\ } \color{red}{\texttt{rkmark}}$

If every letter in this $\texttt{rkmark}$ is moved 9 positions backward, we find another $\texttt{rkmark}$ , as follows:

$\texttt{ma\ } \textcolor{red}{\texttt{rkmark\ }}\texttt{mar\ } \textcolor{red}{\texttt{rkmark}}$

Therefore, we can maintain a triple $(l, r, d)$ indicating that the characters in the interval $[l, r]$ are completely identical to those in the interval $[l - d, r - d]$ .

Then, whenever querying position $x$ , we can continually subtract the corresponding $d$ until $x$ lies within the range of the initial string.

Here is some further explanation.

Code

// ttzytt
#include 
using namespace std;
#define ll long long

struct Seg {
    ll l, r, diff;  // Every point in the range [l, r] has an offset of diff from the preceding segment.
};

int main() {
    int t;
    cin >> t;
    while (t--) {
        int n, c, q;
        cin >> n >> c >> q;
        string str;
        cin >> str;
        vector a(c + 1);
        a[0].l = 0, a[0].r = n - 1;
        for (int i = 1; i <= c; i++) {
            ll l, r;
            cin >> l >> r;
            l--, r--;
            a[i].l = a[i - 1].r + 1;        // The left endpoint follows the right endpoint of the preceding segment.
            a[i].r = a[i].l + (r - l);      // Add length - 1 to obtain the right endpoint.
            a[i].diff = a[i - 1].r - l + 1; 
            /* 
                | First segment | Previous segment | Newly inserted segment |
                                |----------------|  \
                               /        ↑          \
                              / Segment being copied    \
                             l                    a[i - 1].r

            Therefore, the offset is a[i - 1].r - l + 1.
             */
        }
        while (q--) {
            ll x;
            cin >> x;
            x--;
            for (int i = c; i >= 1; i--) {
                if (x < a[i].l)     // If the position x does not belong to the current segment.
                    continue;
                else
                    x -= a[i].diff; // Subtract the offset.
            }
            cout << str[x] << '\n';
        }
    }
}

D. Mark and Lightbulbs

Approach

First, let us simulate the fourth sample:

1 2	000101 010011

$\texttt{000101} \\\downarrow\\\texttt{00}\textcolor{red}{\texttt{1}}\texttt{101}\\\downarrow\\\texttt{0}\textcolor{red}{\texttt{1}}\texttt{1101}\\\downarrow\\\texttt{011}\textcolor{red}{\texttt{0}}\texttt{01}\\\downarrow\\\texttt{01}\textcolor{red}{\texttt{0}}\texttt{001}\\\downarrow\\\texttt{0100}\textcolor{red}{\texttt{1}}\texttt{1}\\$

Note: The positions marked in red indicate changes.

We can see that during this process, we can only lengthen or shorten a segment consisting of $1$ s, such as $\texttt{1}$ or $\texttt{111}$ (of course, viewed the other way around, it can be described as a segment consisting of $0$ s), rather than creating a new “ $1$ segment” out of thin air. This is because only when a $1$ changes to a $0$ , or a $0$ changes to a $1$ , will $s_{i - 1}$ and $s_{i + 1}$ differ, allowing us to change $s_i$ .

Therefore, if strings $s$ and $t$ have different numbers of segments, transforming $s$ into $t$ must be impossible.

We can see that in each operation, we can move the beginning or end of a “ $1$ segment” by one position. We can therefore use this fact to calculate the number of steps required to transform $s$ into $t$ .

That is, for every segment in $s$ and $t$ , we calculate the positions at which the segment begins and ends, then calculate the differences between corresponding segment endpoints in $s$ and $t$ . The sum of these differences is the answer.

How do we determine the beginning and end of a segment? They are simply where $0$ changes to $1$ and where $1$ changes to $0$ . Therefore, we create two arrays, $a$ and $b$ . After reading $s$ and $t$ , we traverse the two strings. Whenever $s_i \ne s_{i + 1}$ , we put $i$ into $a$ (and do the same for $t$ and $b$ ). In this way, $a$ and $b$ store all segment endpoints in the two strings.

Code:

#include 
using namespace std;
#define ll long long
int main() {
    int t;
    cin >> t;
    while (t--) {
        string s, t;
        int n;
        cin >> n >> s >> t;
        vector<int> sdiff, tdiff; // a and b in the solution.
        ll ans = 0;
        if (s.front() != t.front() || s.back() != t.back()) {
            // Because we cannot change s[0] or s[n - 1], the first and last characters of s and t must be the same.
            goto FAIL;
        }

        for (int i = 0; i < s.size() - 1; i++) {
            if (s[i] != s[i + 1]) sdiff.push_back(i);
            // If two adjacent characters differ, this position is an endpoint.
            if (t[i] != t[i + 1]) tdiff.push_back(i);
        }
        if (sdiff.size() != tdiff.size()) {
            goto FAIL;
        } else {
            for (int i = 0; i < sdiff.size(); i++) {
                // Calculate the sum of endpoint differences.
                ans += abs(sdiff[i] - tdiff[i]);
            }
        }

    SUCC:
        cout << ans << '\n';
        continue;
    FAIL:
        cout << "-1\n";
    }
}

[MIT 6.s081] Xv6 Lab 3 (2021): Page Tables Record

2022-07-14T22:57:45.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

Update on 2022/9/14: I recently put the lab code on GitHub. If you need a reference, you can find it here:

https://github.com/ttzytt/xv6-riscv

The different branches contain the different labs.

Note: the basic knowledge related to page tables is discussed in this article, which you can use as a reference.

Lab 3: page tables

Speed up system calls

To accelerate system calls, many operating systems reserve some read-only virtual memory in user space and let the kernel share data there. This reduces repeated transitions between user and kernel mode. We need to use this method to accelerate getpid().

The general idea is to place a process’s PID in shared space when the process is created. When user code queries its PID, it no longer needs an ecall transition to the kernel and avoids the overhead of preserving the execution context.

First, add one page to the user’s virtual memory specifically for data shared with the kernel.

Creating a new mapping from virtual to physical memory requires mappages(), implemented in kernel/vm.c:

// Create PTEs for virtual addresses starting at va that refer to
// physical addresses starting at pa. va and size might not
// be page-aligned. Returns 0 on success, -1 if walk() couldn't
// allocate a needed page-table page.
int
mappages(pagetable_t pagetable, uint64 va, uint64 size, uint64 pa, int perm)
{
  // pagetable is the root page table; va and pa are the starting virtual and physical addresses
  // perm contains the flag bits
  uint64 a, last;
  pte_t *pte;

  if(size == 0)
    panic("mappages: size");
  
  a = PGROUNDDOWN(va);
  last = PGROUNDDOWN(va + size - 1);
  // PGROUNDDOWN effectively sets the final twelve bits of a number to zero.
  // Therefore, a is the start of the new mapping and last is the final page frame to map.

  for(;;){
    if((pte = walk(pagetable, a, 1)) == 0)
      return -1;
    if(*pte & PTE_V)
      panic("mappages: remap");
    *pte = PA2PTE(pa) | perm | PTE_V;
    if(a == last)
      break;
    a += PGSIZE;
    pa += PGSIZE;
    // Allocate one new page each time
  }
  return 0;
}

We can therefore call mappages() from proc_pagetable() in kernel/proc.c to create the additional mapping.

proc_pagetable() is invoked when a new process is created, which meets our requirement.

First, observe how proc_pagetable() uses mappages() to create the trampoline and trapframe pages:

if(mappages(pagetable, TRAMPOLINE, PGSIZE,
            (uint64)trampoline, PTE_R | PTE_X) < 0){
    uvmfree(pagetable, 0);
    return 0;
}

// map the trapframe just below TRAMPOLINE, for trampoline.S.
if(mappages(pagetable, TRAPFRAME, PGSIZE,
            (uint64)(p->trapframe), PTE_R | PTE_W) < 0){
    // On failure, unmap the preceding mapping rather than this nonexistent one
    uvmunmap(pagetable, TRAMPOLINE, 1, 0);
    uvmfree(pagetable, 0);
    return 0;
}

If the current page cannot be mapped, the previously mapped page is removed with uvmunmap() rather than attempting to unmap the failed page itself. The page table is then released with uvmfree().

This is necessary because uvmunmap() requires the page being unmapped to exist. Attempting to unmap a nonexistent mapping crashes—after all, one cannot remove a mapping that was never created.

Because the current page failed to map, we can only use uvmfree() to release memory rather than unmapping that page.

The source of uvmfree() is:

// Free user memory pages,
// then free page-table pages.
void
uvmfree(pagetable_t pagetable, uint64 sz)
{
  if(sz > 0)
    uvmunmap(pagetable, 0, PGROUNDUP(sz)/PGSIZE, 1);
  freewalk(pagetable);
}

When sz is zero, it calls only freewalk, releasing the memory for the entire page-table hierarchy, including all pages that were previously mapped.

Another detail is that before calling freewalk(), we must ensure all mappings have already been removed, which is why uvmunmap() is called first. The implementation of freewalk() makes this clear:

// Recursively free page-table pages.
// All leaf mappings must already have been removed.
void
freewalk(pagetable_t pagetable)
{
  // there are 2^9 = 512 PTEs in a page table.
  for(int i = 0; i < 512; i++){
    pte_t pte = pagetable[i];
    if((pte & PTE_V) && (pte & (PTE_R|PTE_W|PTE_X)) == 0){
      // this PTE points to a lower-level page table.
      uint64 child = PTE2PA(pte);
      freewalk((pagetable_t)child);
      pagetable[i] = 0;
    } else if(pte & PTE_V){ // Important: PTE_V being one means the mapping remains and causes a panic
      panic("freewalk: leaf");
    }
  }
  kfree((void*)pagetable);
}

Using this information, we can write the mapping for USYSCALL, the shared page. USYSCALL lies below the trampoline and trapframe:

if(mappages(pagetable, USYSCALL, PGSIZE, (uint64)(p->usyscall), PTE_R | PTE_U) < 0){
    // After mapping, accessing the page beginning at USYSCALL reaches p->usyscall
    uvmunmap(pagetable, TRAMPOLINE, 1, 0);
    uvmunmap(pagetable, TRAPFRAME, 1, 0);
    uvmfree(pagetable, 0);
    return 0;
  }

Because this page is shared with user mode, both the PTE_R and PTE_U flags must be set. They permit reading and user-mode access, respectively.

As with the earlier calls to mappages(), if mapping fails, first remove the mappings that succeeded earlier and then clear all data belonging to the page table.

After writing this code, accessing an address in the USYSCALL page from user mode reaches the kernel’s p->usyscall storage.

Just as Lab 2 added a trace_mask field to proc, creating an additional page mapping when a process is created means we must remove that mapping when the process is destroyed.

Therefore, modify proc_freepagetable() in kernel/proc.c:

// Free a process's page table, and free the
// physical memory it refers to.
void
proc_freepagetable(pagetable_t pagetable, uint64 sz)
{
  uvmunmap(pagetable, USYSCALL, 1, 0); // Newly added
  uvmunmap(pagetable, TRAMPOLINE, 1, 0);
  uvmunmap(pagetable, TRAPFRAME, 1, 0);
  uvmfree(pagetable, sz);
}

One problem remains. We have created a virtual-to-physical mapping, but have not allocated the corresponding physical memory when creating the process. Without allocating it, we would try to map virtual memory to a null pointer, which naturally causes a failure.

We therefore also need to modify allocproc().

Observe how allocproc() allocates physical memory for the trapframe:

if((p->trapframe = (struct trapframe *)kalloc()) == 0){
    freeproc(p);
    release(&p->lock);
    return 0;
}

The logic is straightforward, so we can directly use it as a reference.

// Allocate the usyscall page
if((p->usyscall = (struct usyscall *)kalloc()) == 0){
    freeproc(p->usyscall);
    release(&p->lock);
    return 0;
}
p->usyscall->pid = p->pid;
// Store the PID immediately after creating it

The kernel-side work is now complete. We do not need to write the user-mode function ourselves because, as the lab hint says, it is already implemented in user\ulib.c:

int
ugetpid(void)
{
  struct usyscall *u = (struct usyscall *)USYSCALL;
  return u->pid;
}

As described earlier, directly accessing the USYSCALL virtual address reaches the contents stored at the physical address p->usyscall. Strictly speaking, that kernel address is also virtual, but most kernel virtual addresses are directly mapped to physical addresses.

This completes the task.

Print a page table

Implement a vmprint() function. It accepts a pagetable_t and prints the page table in the format shown in the image. Call this function to print the page table when creating the init process.

Ignore the call during init creation for the moment and first implement the function in kernel/vm.c.

xv6 uses a multilevel page table, so its structure is a tree. If this is unfamiliar, see this article. In essence, we need a DFS that prints a tree.

The implementation is:

void 
vmprint(pagetable_t pagetable, uint dep){
  if(dep == 0)
    printf("page table %p\n", pagetable);
  for(int i = 0; i < 512; i++){
    pte_t pte = pagetable[i];
    if(pte & PTE_V){
      for(int j = 0; j < dep; j++)
        printf(".. ");
      uint64 child = PTE2PA(pte);
      printf("..%d: pte %p pa %p\n", i, pte, child);
      if(dep < 2)
        // At depth 2, stop recursing because this is a leaf
        vmprint((pagetable_t) child, dep + 1);
    }
  } 
}

This function accepts two arguments: the page table to print, which can be understood as the root of the tree, and the current depth. The depth is needed because the required format prints a different number of dots at each level. It also tells us whether a leaf has been reached.

Each pagetable contains at most 512 entries, so traverse them in order. If an entry is allocated, meaning that pte & PTE_V is nonzero, continue recursively.

Before printing each entry, output dep + 1 groups of .., followed by its PTE and PA.

Here, PTE means the value read directly from the page-table entry. PA is the physical address after removing the flag bits from that entry. The physical address leads to either the next-level page table or a page frame.

The expression pte_t pte = pagetable[i]; works because PA points to the first element of the child page table, and pagetable[i] is equivalent to *(pagetable + i), which accesses page-table entry $i$ .

This completes the main part. Insert the following near the end of kernel/exec.c:

1 2	if(p->pid == 1) vmprint(p->pagetable, 0);

Because init is the first process created by the system, its PID is 1. Its page table will therefore be printed when init is created.

That completes this part.

Detecting which pages have been accessed

Implement pgaccess(), declared as int pgaccess(void *base, int len, void *mask);. It determines whether pages have been accessed since the previous invocation of this function. base identifies the first page to inspect, len gives the number of pages beginning there, and the access state of every page must be written to mask. This mask works like trace_mask in Lab 2: if a page was accessed, its corresponding bit is one.

Unlike Lab 2, the purpose here is not to learn the system-call registration process. This call has already been registered, so we do not need to repeat that work.

We can directly implement it in kernel/sysproc.c.

The first step is necessarily to obtain the user-supplied arguments with the arg family of functions. The reason is explained in the Lab 2 article. This gives the following code:

pagetable_t u_pt = myproc()->pagetable;
uint64 fir_addr, mask_addr;
uint ck_siz; 
uint mask = 0;
try(argaddr(0, &fir_addr), return -1);
try(argint(1, &ck_siz), return -1);
try(argaddr(2, &mask_addr), return -1);

Here, fir_addr, ck_siz, and mask_addr correspond to the three declared arguments.

Next, consider how to determine whether a page has been accessed. We use flag bits in the PTE, explained in the xv6 study notes. The RISC-V specification says:^[1]

Each leaf PTE contains an accessed (A) and dirty (D) bit. The A bit indicates the virtual page has been read, written, or fetched from since the last time the A bit was cleared. The D bit indicates the virtual page has been written since the last time the D bit was cleared.
Translation: every leaf PTE has accessed (A) and dirty (D) flags. A records whether the virtual address has been read, written, or used since A was last reset. D records whether the virtual address has been written since D was last reset.

These flags are set by the RISC-V processor and require no software action. The function only needs to read and reset them.

Because we need to detect any access rather than only writes, we use the A flag. xv6 does not yet define PTE_A, so add it to kernel/riscv.h:

1	#define PTE_A (1L << 6) // The diagram above shows that it is shifted by six bits

Then write the following in sys_pgaccess:

if(ck_siz > 32){
    return -1;
}

pte_t* fir_pte = walk(u_pt, fir_addr, 0);

for(int i = 0; i < ck_siz; i++){
    if((fir_pte[i] & PTE_A) && (fir_pte[i] & PTE_V)){
        mask |= (1 << i);
        fir_pte[i] ^= PTE_A; // Reset
    }
}

If ck_siz is greater than 32, the mask does not contain enough bits to store the results, so the function must return an error.

The walk() function below is important. I will not explain its detailed implementation here. Given a page table and virtual address, walk() returns the leaf PTE corresponding to that virtual address.

It therefore gives us fir_pte, the address of the PTE for the first page to inspect.

Next, inspect the PTE_A flag in the following ck_siz entries:

for(int i = 0; i < ck_siz; i++){
    if((fir_pte[i] & PTE_A) && (fir_pte[i] & PTE_V)){
        mask |= (1 << i);
        fir_pte[i] ^= PTE_A; // Reset
    }
}

Finally, return the computed mask to user mode using copyout(), which is explained in the Lab 2 article.

In brief, given a user page table and virtual address, copyout() copies data from kernel mode to that location in user mode.

We can therefore write:

1	try(copyout(u_pt, (uint* )mask_addr, &mask, sizeof(uint)), return -1);

This copies the mask data to mask_addr, interpreted using the user-mode page table.

The lab is now complete.

Summary

The concepts of page tables and virtual addresses are honestly more difficult than system calls. Completing this lab requires a clear understanding of the RISC-V page-table implementation, and it took me a long time to understand it. Only after doing the lab did I appreciate how ingenious the design of page tables and virtual addresses is.

I wish everyone working on this lab an early AC:

1.https://github.com/riscv/riscv-isa-manual/releases/download/Ratified-IMFDQC-and-Priv-v1.11/riscv-privileged-20190608.pdf ↩

CF1703 F, G Solutions

2022-07-12T18:45:57.000Z

The content below was generated entirely by machine translation. Please verify its accuracy. If anything is unclear, consult the Chinese source version.

F. Yet Another Problem About Pairs Satisfying an Inequality

Approach (from the Official Solution)

Observe the inequality $a_i < i < a_j < j$ in the problem. We can see that, for any element in the array, if it does not satisfy $a_x < x$ , then it can never form a valid pair with any element. Therefore, we can directly skip elements that do not satisfy $a_x < x$ .

We can split this inequality into three parts: $a_i < i$ , $i < a_j$ , and $a_j < j$ .

For all elements satisfying $a_x < x$ , the first and last inequalities are already satisfied. We only need to find elements satisfying $i < a_j$ to form a valid pair.

Let $b$ be the array obtained by removing from $a$ the elements that do not satisfy $a_x < x$ (this sounds a little strange, but the index of every element in $b$ is the same as it was in $a$ ).

For example, suppose an element in the array has two properties, value and index, and we denote it as $(val, id)$ . If array $a$ is:

$a = (1, 1) \ (1, 2)\ (2, 3)\ (3, 4)\ (8, 5)\ (2, 6)\ (1, 7)\ (4, 8)\\$

then removing all elements with $val >= id$ gives $b$ :

$b = (1, 2)\ (2, 3)\ (3, 4)\ (2, 6)\ (1, 7)\ (4, 8)\\$

Then we can solve the problem by finding all valid $i$ for every $b_j$ .

It is not difficult to see that $i$ increases monotonically, so binary search can be used to find the largest $i$ smaller than $b_j$ . Every element in $b$ whose $id$ is smaller than $i$ (as well as $i$ itself) can form a valid pair with $a_j$ .

In addition to binary search, we can use a Fenwick tree to find the number of elements in $b$ whose $id$ is smaller than a particular value.

Specifically, we can use a Fenwick tree to maintain a prefix-sum array and then traverse the elements in $b$ , performing upd(id) each time. This makes the value found when querying every number greater than or equal to $id$ increase by $1$ .

Thus, querying $b_j - 1$ in the Fenwick tree returns all $i$ smaller than $b_j$ .

Of course, a difference-array method can also produce the same prefix-sum array as the Fenwick tree. This problem does not require any further updates after obtaining the prefix-sum array, so a difference array can solve the problem in $\operatorname{O}(n)$ time.

Code

Complexity: $\operatorname{O}(n \log n)$

#include 
using namespace std;
#define ll long long
// author: ttzytt (ttzytt.com)
// ref: https://codeforces.com/blog/entry/104786
int main() {
    int t;
    cin >> t;
    while (t--) {
        int n;
        cin>>n;
        int a[n + 1];
        for (int i = 1; i <= n; i++) {
            cin >> a[i];
        }
        ll ans = 0;
        vector<int> valid; // The array b described above, but storing only indices,
                           // because we only need to find the largest index j smaller than b_i.
        for (int i = 1; i <= n; i++) {
            if (a[i] >= i) continue; // Skip it directly if it does not satisfy the condition.
            
            // This may count as an optimization. We can see that every index i in valid is smaller than j.
            // We do not put all indices of b into valid, because a[i] < i < a[j] < j,
            // so the condition can only hold when i < j.
            ans += (ll)(lower_bound(valid.begin(), valid.end(), a[i]) -
                        valid.begin());
            // lower_bound finds the first element in valid that is greater than or equal to a[i].
            // Therefore, every element **before** it can be used. The length of an interval is r - l + 1.
            // Since only the elements before it can be used, we do not add this 1.
            valid.push_back(i);
        }
        cout << ans << '\n';
    }
}

G. Good Key, Bad Key

Approach (from the Official Solution)

We can see that alternating between good keys and bad keys for all boxes is always less worthwhile.

Moreover, using good keys consecutively on the earlier boxes is more worthwhile. (In other words, use good keys on a prefix.)

Suppose we use a bad key before a good key. Then the profit we obtain is:

$\lfloor \frac{a_i}{2} \rfloor + (\lfloor \frac{a_{i + 1}}{2} \rfloor - k)$

But if we first use a good key and then a bad key, the profit is:

$(a_i - k) + \lfloor \frac{a_{i + 1}}{2} \rfloor$

Clearly, using the good key first is more worthwhile.

A more intuitive explanation is that regardless of which key is used first, $k$ is subtracted from the profit. However, if we use the bad key first, the profits of two boxes are halved, whereas if we use the good key first, only the profit of one box is halved.

Therefore, we only use bad keys in the final part. In some cases where $k$ is relatively large, halving $a_i$ may be more worthwhile than subtracting $k$ .

Thus, we only need to enumerate a dividing point between the use of good and bad keys. Use good keys before this point and bad keys after it.

Let this dividing point be $x$ .

When bad keys are used after the dividing point $x$ , the profit from each box becomes:

$a_{x + i} = \lfloor a_{x + i} \div 2^{i} \rfloor$

We can see that this $2^i$ grows very quickly, and beyond some point $a_i$ becomes $0$ . There is no need to continue calculating beyond that point.

Because the maximum $a_i$ is $10^9$ , there is no need to calculate after $log_2(10^9) \approx 30$ . (Alternatively, consider continually shifting a number to the right; after some point, its entire binary representation contains no $1$ s.)

Code

#include 
using namespace std;
#define ll long long
// author: ttzytt (ttzytt.com)
// ref: https://codeforces.com/blog/entry/104786
int main() {
    int t;
    cin >> t;
    while (t--) {
        int n, k;
        cin >> n >> k;
        vector<int> a(n);
        for (int i = 0; i < n; i++) {
            cin >> a[i];
        }
        ll psum = 0, ans = 0;
        // psum is the profit obtained by using good keys.
        for (int i = -1; i < n; i++) {
            if (i != -1) psum += (ll)(a[i] - k);
            ll cur = psum;
            // Enumerate i as the dividing point.
            for (int j = i + 1; j < min(n, i + 32); j++) {
                // There is no need to continue calculating beyond i + 32.
                int bkval = a[j];
                bkval >>= (j - i);  // i + 1 is divided by 2, i + 2 is divided by 4, ...
                cur += bkval;
            }
            ans = max(ans, cur);
        }
        cout << ans << endl;
    }
}

tzyt's Blog

Understanding the Least-Squares Method From the Perspectives of Calculus and Linear Algebra

Definition of the Problem

Calculus Perspective

Linear Algebra Perspective

GAMES101 Study Notes 1

Three-Dimensional Rotation Matrices

Barycentric Coordinates

Physical Perspective

Algebraic Perspective

USACO23JAN Find and Replace S (Luogu P9013) Solution

Analysis

Handling Cycles

Two Special Cases

1

2

Code Implementation

[Stanford CS144] Lab 4 Record

Implementing the TCP State-Transition Diagram

Establishing a TCP Connection

Active Connection

Passive Connection

Closing a TCP Connection

Passive Close

Active Close

Implementing Passive Close

Implementing Active Close

[Stanford CS144] Lab 0–Lab 3 Lab Notes

Lab 0: Networking Warmup

Lab 1: Stitching Substrings into a Byte Stream

Implementation

Rough Description

Example

Requirements

Algorithm

Lab 2: The TCP Receiver

Sequence-Number Wrapper

CF1774C Solution

Approach

Special Case

Generalization

Code and Implementation

Derivation and Implementation of a Dense Neural Network on MNIST

Formula Derivation

Notation and Language Conventions

A Neural Network with One Node per Layer

Multiple Nodes per Layer

Partial Derivative of the Error with Respect to a Weight

Partial Derivative of the Error with Respect to a Bias

Partial Derivative of the Error with Respect to the Previous Layer’s Input

Implementation

Data Preprocessing

The layer Class

The neu_net Class

Initialization Function

Output Function

Backpropagation

Results

Ray Tracing: The Next Week Study Notes (1)

bvh_node

Spherical Texture Coordinates

Checkerboard Texture

Implementation in the Book

Checkerboard Only on the Surface

Perlin Noise

Turbulence

Some Questions

Instance Transformations

Rotation Matrices

Formula Derivation

Some Explanations

Some Small Implementation Issues

Volumetric Fog

Multithreading

Ray Tracing in One Weekend Study Notes (2): Implementing the Camera Class

Implementing the Camera Class

Positioning the Camera

Implementing Depth of Field

Depth of Field in Reality

Actual Implementation

The `layer` Class

The `neu_net` Class

`bvh_node`