Async/await in C#. Internals, useful tricks and features. Little known and interesting facts

Hi. This time we’ll talk about a topic that every self-respecting adherent of the C# language has tried to understand — asynchronous programming using Tasks or async/await. Microsoft did a great job — in order to use Tasks with async/await in most cases you only need to know the syntax and nothing else. But if you go deeper, the topic is quite voluminous and complex. There are a lot of cool articles on this topic, but there are still a lot of misconceptions around it. We will try to correct the situation and chew the material as much as possible, without sacrificing either depth or understanding.

Topics / chapters covered:

1. Conception of asynchrony — benefits of asynchrony and «blocked» thread myths
2. TAP. Syntax and compilation conditions — prerequisites for writing a correct method
3. Coding using TAP — the mechanics and behavior of the program in asynchronous code (freeing threads, starting tasks and waiting for them to complete)
4. Behind the scenes: The State Machine — overview of compiler transformations and classes generated by it
5. The origins of asynchrony. The device standard asynchronous methods — asynchronous methods for working with files and the network from the inside
6. Classes and tricks with TAP — useful tricks that can help with managing and speeding up a program using TAP

Conception of asynchrony

Asynchrony is far from new. Typically, asynchrony means performing an operation in a style that does not imply blocking the calling thread, that is, starting the operation without waiting for it to complete. Blocking is not as evil as it is described. One may come across claims that blocked threads waste CPU time, work more slowly and cause rain. Does the latter seem unlikely? Actually the previous 2 points are almost the same.

At the OS scheduler level, when a thread is in a “blocked” state, precious processor time slice will not be given to it. Scheduler calls, as a rule, fall on operations that cause blocking, timer interrupts, and other interrupts. That is, when, for example, the disk controller completes the read operation and initiates an appropriate interrupt, the scheduler starts. Scheduler will decide whether to start a thread that was unblocked by this operation, or some other one with a higher priority.

Slow work seems even more absurd. In fact, on low level the work is the same. But there are additional small overheads for performing the asynchronous operation. 
But of course, enterprise apps get lots of advantages from asyncs, so at high level it can be treated as "slow work of blocking code". And it will be true. I'm just showing you it from another point, to make your mind strong.

The causing of rain is generally not something from this area.
The main blocking problem is the unreasonable consumption of computer resources. Even if we forget about the time to create thread, every blocked thread consumes extra space. Also, there are scenarios where only one thread can perform certain work (for example, a UI thread). Accordingly, I wouldn't want him to be busy with a task that another thread can perform, sacrificing the performance of exclusive operations for him.

Asynchrony is a very broad concept and can be achieved in many ways.

In the history of .NET, there were the following concepts:

1. EAP (Event-based Asynchronous Pattern) — as the name implies, the approach is based on events that fire when the operation completes. The usual method invokes the operation and subscribe some callback/continuation on event
2. APM (Asynchronous Programming Model) — based on 2 methods. The BeginSmth method returns the IAsyncResult interface. The EndSmth method accepts IAsyncResult (if the operation is not completed by the time EndSmth is called, the thread is blocked)
3. TAP (Task-based Asynchronous Pattern) — that is async/await (strictly speaking, these words appeared after birth of the Task and Task<TResult>, but async/await greatly improves this concept)

The last approach was so successful that everyone successfully forgot about the previous ones. So, the article will be about him.

Task-based asynchronous pattern. Syntax and compilation conditions

The standard TAP-style asynchronous method is very simple to write.

It needs:

1. The type of the return value to be Task, Task<T> or void (not recommended, explained further). In C# 7 came Task-like types (discussed in the last chapter). In C# 8 IAsyncEnumerable<T> and IAsyncEnumerator<T> are added to this list
2. The method to be marked with the async keyword and to contain await inside. These keywords are paired. Moreover, if the method contains await, be sure to mark it async, the opposite is not true, but it is useless
3. For decency, abide by the Async suffix convention. Of course, the compiler will not consider this an error. If you are a very decent developer, you can add overloads with the CancellationToken (discussed in the last chapter) or even IProgress, lol

For such methods, the compiler does a serious job. And they become completely unrecognizable behind the scenes, but more on that later.

It was mentioned that the method should contain the await keyword. It indicates the need for asynchronous waiting for the task to be completed. This task is represented by Task type and it instances. Await keyword is applied exactly to that instance.

The Task instance also has certain conditions so that await can be applied to it:

1. The awaited type (Task, mentioned previously) must have a public (or internal) GetAwaiter() method, it can be an extension method. This method returns an awaiter.
2. The awaiter must implement the INotifyCompletion interface, which requires the implementation of the void OnCompleted(Action continuation) method. It should also have the instance property bool IsCompleted and GetResult() method. It can be either a structure or a class.

The example below shows how to make an int awaitable, and even never executed.

public class Program
{
    public static async Task Main()
    {
        await 1;
    }
}

public static class WeirdExtensions
{
    public static AnyTypeAwaiter GetAwaiter(this int number) => new AnyTypeAwaiter();

    public class AnyTypeAwaiter : INotifyCompletion
    {
        public bool IsCompleted => false;

        public void OnCompleted(Action continuation) { }

        public void GetResult() { }
    }
}

Coding using TAP

It is difficult to go into the internals without understanding how something should work. Consider TAP in terms of program behavior.

Terminology: the considered asynchronous method, whose code will be considered, I will call the asynchronous method, and the called asynchronous methods inside it I will call the asynchronous operation.

Example. As an asynchronous operation we take Task.Delay, which delays for the specified time without blocking the thread.

public static async Task DelayOperationAsync() // asynchronous method
{
    BeforeCall();
    Task task = Task.Delay(1000); //asynchronous operation
    AfterCall();
    await task;
    AfterAwait();
} 

The implementation of the method in terms of behavior is as follows.

1. All code that precedes the invocation of the asynchronous operation is executed. In this case, this is the BeforeCall method
2. An asynchronous operation is called. At this stage, the thread is not free or blocked. This operation returns the result — the mentioned Task, which is stored in a local variable
3. The code after the asynchronous operation, but before waiting (await) is executed. In the example — AfterCall
4. Waiting for completion on a task object (which is stored in a local variable) — await task. If the asynchronous operation is completed at this point, then execution continues synchronously, in the same thread. If the asynchronous operation is not completed, then the code that should be executed after completion of the asynchronous operation. The continuation (everything after await, that should be executed after) is "saved", and the thread returns to the thread pool and becomes available for use.
5. Execution of the code after waiting — AfterAwait — is performed either immediately, in the same thread when the operation at the time of waiting was completed, or, upon completion of the operation, a new thread is taken to continue execution("saved" continuation in the previous step)

Behind the scenes. State machine

In fact, our method is transformed by the compiler into a stub method in which the generated class — the state machine — is initialized. Then it (the machine) starts up, and the task object used in step 2 is returned from the method.

Of particular interest is the MoveNext method of the state machine. This method does everything the asynchronous method does before compiler transformation. It breaks the code between each await call. Each part is executed in a certain condition of the machine. The MoveNext method itself attaches to the task object as a continuation. Saving the state guarantees the execution of precisely that part of the method that logically follows the awaiting.

As the Russian proverb says, it’s better to see once than to hear 100 times, so I highly recommend that you familiarize yourself with the example below. I rewrote the code a bit, improved variable naming, and commented generously.

Source code

public static async Task Delays()
{
     Console.WriteLine(1);
     await Task.Delay(1000);
     Console.WriteLine(2);
     await Task.Delay(1000);
     Console.WriteLine(3);
     await Task.Delay(1000);
     Console.WriteLine(4);
     await Task.Delay(1000);
     Console.WriteLine(5);
     await Task.Delay(1000);
}

Stub method

[AsyncStateMachine(typeof(DelaysStateMachine))]
[DebuggerStepThrough]
public Task Delays()
{
     DelaysStateMachine stateMachine = new DelaysStateMachine();
     stateMachine.taskMethodBuilder = AsyncTaskMethodBuilder.Create();
     stateMachine.currentState = -1;
     AsyncTaskMethodBuilder builder = stateMachine.taskMethodBuilder;
     taskMethodBuilder.Start(ref stateMachine);
     return stateMachine.taskMethodBuilder.Task;
}

State machine

[CompilerGenerated]
private sealed class DelaysStateMachine : IAsyncStateMachine
{
    //reflects the current state, indicates the await on which we are waiting
    //that is for possibility to restore method execution from any await
    public int currentState; 
    public AsyncTaskMethodBuilder taskMethodBuilder;
    //current awaiter
    private TaskAwaiter taskAwaiter;

    //all parameters of the method, as well as local variables, are stored in     fields in order to save between waitings when the thread is "released" and     returned to the thread pool
    public int paramInt;
    private int localInt;

    private void MoveNext()
    {
        int num = currentState;
        try
        {
            TaskAwaiter awaiter5;
            TaskAwaiter awaiter4;
            TaskAwaiter awaiter3;
            TaskAwaiter awaiter2;
            TaskAwaiter awaiter;
            switch (num)
            {
                default:
                    localInt = paramInt;  //before first await
                    Console.WriteLine(1);  //before first await
                    awaiter5 = Task.Delay(1000).GetAwaiter();  //before first await
                    if (!awaiter5.IsCompleted) // await itself. Here is the check, that determine whether the async operation is completed to this moment
                    {
                        num = (currentState = 0); //state update to continue with right place
                        taskAwaiter = awaiter5; //copy awaiter to field in order to save it between thread releases and continuations
                        DelaysStateMachine stateMachine = this; //it is needed to pass this by ref
                        taskMethodBuilder.AwaitUnsafeOnCompleted(ref awaiter5, ref stateMachine); //behind the scary name are actions on joining the continuation to wait object. Continuation is current method itself (MoveNext)
                        return;
                    }
                    goto Il_AfterFirstAwait; //if the method is completed, go to the code that was right after this
                case 0: //this state will be only when the first asynchronous operation did not complete by the time of check (if (!awaiter5.IsCompleted) ), so the method went along the asynchronous execution path. If we are here, then the first operation is completed and we resume execution of the method, it is continuation
                    awaiter5 = taskAwaiter; //restore the wait object
                    taskAwaiter = default(TaskAwaiter); //clear the field with wait object
                    num = (currentState = -1); //state update
                    goto Il_AfterFirstAwait; //go directly to the logic from the original method
                case 1: // we are here, if the second operation did not end synchronously, and the continuation was joined, which just started.
                    awaiter4 = taskAwaiter;
                    taskAwaiter = default(TaskAwaiter);
                    num = (currentState = -1);
                    goto Il_AfterSecondAwait;
                case 2: //likewise, the third operation did not complete immediately.
                    awaiter3 = taskAwaiter;
                    taskAwaiter = default(TaskAwaiter);
                    num = (currentState = -1);
                    goto Il_AfterThirdAwait;
                case 3: // а здесь четвертая
                    awaiter2 = taskAwaiter;
                    taskAwaiter = default(TaskAwaiter);
                    num = (currentState = -1);
                    goto Il_AfterFourthAwait;
                case 4: // ну и пятая
                    {
                        awaiter = taskAwaiter;
                        taskAwaiter = default(TaskAwaiter);
                        num = (currentState = -1);
                        break;
                    }

                    Il_AfterFourthAwait:
                    awaiter2.GetResult();
                    Console.WriteLine(5); //code after the fourth asynchronous operation
                    awaiter = Task.Delay(1000).GetAwaiter(); //fifth asynchronous operation
                    if (!awaiter.IsCompleted)
                    {
                        num = (currentState = 4);
                        taskAwaiter = awaiter;
                        DelaysStateMachine stateMachine = this;
                        taskMethodBuilder.AwaitUnsafeOnCompleted(ref awaiter, ref stateMachine);
                        return;
                    }
                    break;

                    Il_AfterFirstAwait: //if we are here, then the first operation ended one way or another (sync or async)
                    awaiter5.GetResult(); //accordingly, the result is available and we got it
                    Console.WriteLine(2); //execution of the code that came after the first await
                    awaiter4 = Task.Delay(1000).GetAwaiter(); //Second asynchronous operation
                    if (!awaiter4.IsCompleted) 
                    {
                        num = (currentState = 1);
                        taskAwaiter = awaiter4;
                        DelaysStateMachine stateMachine = this;
                        taskMethodBuilder.AwaitUnsafeOnCompleted(ref awaiter4, ref stateMachine);
                        return;
                    }
                    goto Il_AfterSecondAwait;

                    Il_AfterThirdAwait:
                    awaiter3.GetResult();
                    Console.WriteLine(4); //code after the third asynchronous operation
                    awaiter2 = Task.Delay(1000).GetAwaiter(); //fourth asynchronous operation
                    if (!awaiter2.IsCompleted)
                    {
                        num = (currentState = 3);
                        taskAwaiter = awaiter2;
                        DelaysStateMachine stateMachine = this;
                        taskMethodBuilder.AwaitUnsafeOnCompleted(ref awaiter2, ref stateMachine);
                        return;
                    }
                    goto Il_AfterFourthAwait;

                    Il_AfterSecondAwait:
                    awaiter4.GetResult();
                    Console.WriteLine(3); //code after the second asynchronous operation
                    awaiter3 = Task.Delay(1000).GetAwaiter(); //third asynchronous operation
                    if (!awaiter3.IsCompleted)
                    {
                        num = (currentState = 2);
                        taskAwaiter = awaiter3;
                        DelaysStateMachine stateMachine = this;
                        taskMethodBuilder.AwaitUnsafeOnCompleted(ref awaiter3, ref stateMachine);
                        return;
                    }
                    goto Il_AfterThirdAwait;
            }
            awaiter.GetResult();
        }
        catch (Exception exception)
        {
            currentState = -2;
            taskMethodBuilder.SetException(exception);
            return;
        }
        currentState = -2;
        taskMethodBuilder.SetResult(); //if our method return a result, it would be a generic parameter of the SetResult<T>()
    }

    void IAsyncStateMachine.MoveNext() {...}

    [DebuggerHidden]
    private void SetStateMachine(IAsyncStateMachine stateMachine) {...}

    void IAsyncStateMachine.SetStateMachine(IAsyncStateMachine stateMachine) {...}
}

Pay attention to the phrase «determine whether the async operation is completed to this moment.» An asynchronous operation can also follow a synchronous execution path. The main condition for the current asynchronous method to be executed synchronously (that is, without changing the thread) is the completion of the asynchronous operation at the time of check for IsCompleted.

This listening demonstrates this behavior

static async Task Main()
{
    Console.WriteLine(Thread.CurrentThread.ManagedThreadId); //1
    Task task = Task.Delay(1000);
    Thread.Sleep(1700);
    await task;
    Console.WriteLine(Thread.CurrentThread.ManagedThreadId); //1
}

About the synchronization context. AwaitUnsafeOnCompleted method that is used in state machine, in the end calls Task.SetContinuationForAwait method. In this method the current synchronization context is got with SynchronizationContext.Current property. Synchronization context can be treat as the type of the thread. If synchronization context exists (e.g. UI thread context), continuation is created with SynchronizationContextAwaitTaskContinuation class. This class calls Post method on the saved context to execute the continuation. This guarantees that continuation will be executed exactly in the same context, where the method was started. The specific logic for executing the continuation depends on the Post method in a context that. If there was no synchronization context (or it was specified that it doesn't matter that context will execute the continuation), the continuation will be executed by thread from thread pool. We can specify that it doesn't matter that context will execute the continuation with ConfigureAwait(false) method, which will be discussed in the last chapter.

The origins of asynchrony. The device standard asynchronous methods

We looked at how a method using async/await looks and what happens behind the scenes. But it’s important to understand the nature of asynchronous operations. As we see, asynchronous methods contain asynchronous operations. However, what happens inside the asynchronous operations themselves? Probably the same, but this cannot happen ad infinitum.

It is very important to understand the nature of asynchrony. When trying to understand asynchrony, there is an alternation of states «everything clear» and «can't understand anything» And this alternation will be until the source of asynchrony is understood.

When working with asynchrony, we operate on tasks. This is not at all the same as a thread. One task can be executed by many threads, and one thread can execute many tasks.

Asynchrony usually starts with a method that returns Task (for example), but is not marked with async, and accordingly does not use await inside. This method is not modofied by the compiler; it is executed as is.

So, let's look at some of the roots of asynchrony.

1. Task.Run, new Task(..).Start(), Factory.StartNew and so forth. The easiest way to start async execution. These methods simply create a new task object, passing a delegate as one of the parameters. The task is transferred to the scheduler, which makes all decisions and select thread from thread pool(almost always) to execute it. These methods return ready task object, that can be awaited. Typically, this approach is used to start CPU-bound operations in a separate thread.
2. TaskCompletionSource. A helper class that controls the task object. Designed for those who cannot pick out a delegate to use in mentioned above methods and use more sophisticated mechanisms for controlling completion. It has a very simple API — SetResult, SetError, etc., which update the task accordingly (marking as completed and setting result or error). This task is available through the Task property of the TaskCompletionSource. Perhaps inside you will create threads, have complex logic for their interaction or completion by event. A little more details about this class will be in the last section.

Also the methods of standard libraries may be mentioned like additional paragraph. These include reading / writing files, working with the network, and so on. As a rule, such popular and common methods use system calls that vary on different platforms. And their device is extremely entertaining. Consider working with files and the network.

Files

An important note — if you want to asynchronously work with files, you must specify useAsync = true when creating FileStream.

Files internals are non-trivially and confusingly. The FileStream class is declared as partial. And besides it, there are 6 more platform-specific add-ons. So, in Unix, asynchronous access to file, as a rule, uses a synchronous operation in a separate thread. In Windows, there are system calls for asynchronous operation, which, of course, are used. This leads to differences in work on different platforms. Sources.

Unix

The standard behavior when writing or reading is to perform the operation synchronously, if the buffer allows and the stream is not busy with another operation. Let's explain these two conditions.

1. Stream is not busy with another operation

The Filestream class has an object inherited from SemaphoreSlim with the parameters (1, 1) — that is, a la critical section — the code fragment protected by this semaphore can be executed by only one thread at a time. This semaphore is used both for reading and writing. That is, it is impossible to simultaneously produce both reading and writing. And «classical» blocking on the semaphore does not occur. Call to this._asyncState.WaitAsync() method is performed on it, which returns the task object (there is no lock at all, it would be if the await keyword were applied to the result of the method). If this task object is not completed — that is, the semaphore is captured by another thread, then the continuation in which the operation is performed is attached to the returned wait object (Task.ContinueWith). If the object is free, then you need to check the second mentioned condition.

2. Buffer allows (the second mentioned condition)

Here the behavior depends on the nature of the operation.

For writing — if (the size of the data for writing + position in the file) is less than the size of the buffer, which by default is 4096 bytes, the data is written to buffer synchronously. That is, we must write 4096 bytes from the beginning, 2048 bytes with an offset of 2048, and so on. If this is the case, then the operation is carried out synchronously (written to buffer, not directly to file), otherwise the continuation is attached (Task.ContinueWith). The continuation uses a regular synchronous system call, but the continuation is executed by ThreadPool thread (in most cases), so the whole operation from .NET perspective is async.
For reading — it is checked whether there is enough data in the buffer in order to return all the necessary data. If not, then, again, a continuation (Task.ContinueWith) with a synchronous system call.

By the way, there is an interesting detail here. If one piece of data takes up the entire buffer, it will be written directly to the file, without the participation of the buffer. So, if we create a stream and write 4097 bytes into it, then they will immediately appear in the file, even without calling Dispose (for flushing). If we write 4095, then there will be nothing in the file. At the same time, there is a situation when there will be more data than the size of the buffer, but they will all pass through it (although as mentioned earlier, all data that is greater than buffer size is written directly to the file). This happens if there is already something in the buffer. Then our data will be divided into 2 parts, one will fill the buffer to the end and the data will be written to the file, the second will be written to the buffer if it fits it or directly to the file if it does not.

Windows

In Windows, the algorithm for using the buffer and writing directly is very similar. But a significant difference is observed directly in asynchronous write and read system calls. If we speak without delving into system calls, then there is such a structure Overlapped. It has an important field for us — HANDLE hEvent. This is a manual reset event that becomes signaled upon completion of the operation. Back to the implementation. Writing directly, like writing a buffer, uses asynchronous system calls that use the above structure as a parameter. When writing, a FileStreamCompletionSource object is created — it derived from TaskCompletionSource, in which IOCallback is specified. It is called by a free thread from the pool when the operation completes. In the callback, the Overlapped structure is parsed and the Task object is updated accordingly. That's all the magic.

Network

It is difficult to describe everything that I saw while looking at the source code. My path was from HttpClient to Socket and SocketAsyncContext for Unix. The general scheme is the same as with files. For Windows, the above-mentioned Overlapped structure is used and the operation is performed asynchronously. 

Async in operating system

And a little explanation. The attentive reader will notice that when using asynchronous calls between the call and the callback, there is some void that somehow works with the data. It is worth clarifying here for the sake of completeness. Using files as an example, the disk controller directly performs computational operations with the disk, he gives signals about the movement of the heads to the desired sector, and so on. The processor is free at this time. Communication with the disk occurs through the I/O ports. They contains the type of operation, the location of the data on the disk, and so on. Further, the controller and the disk are engaged in performing this operation, and upon completion of the work, they generate an interrupt. Accordingly, an asynchronous system call only adds information to the I/O ports, while a synchronous system call also waits for the results, putting the thread into a blocking state. This scheme does not claim to be absolutely accurate (not about this article), but gives a conceptual understanding of the work.

The end of asyncs

The nature of the process is now clear. But someone may have a question, what to do with asynchrony? It's impossible to write async on a method forever.

Firstly. The application can be made as a service. In this case, the entry point is written from scratch by you. Until recently, Main could not be asynchronous; in version 7 of the language, this feature was added. But it does not change anything fundamentally, the compiler just generates the usual Main, and the asynchronous method is simply a static method that is called in Main and is synchronously expected to complete. So, most likely you have some kind of long-term action. For some reason, at this moment, many begin to think about how to create threads for this business: via Tasks or via Threads manually. The answer is simple — of course Task. If you are using the TAP approach, then you do not need to interfere with manual thread creation. This is akin to using the HttpClient for almost all requests, and doing the POST manually forming request and sending serialized bytes with TCP ports.

Secondly. Web applications. Each request that comes in spawns a new thread being pulled from the ThreadPool for processing. The pool is large, of course, but not endless. In the case when there are many requests, there may not be enough threads for all, and all new requests will be queued for processing. But in the case of using asynchronous controllers, as discussed earlier, the thread is returned to the pool(when it is blocked on any operation) and can be used to process new requests. This significantly increases the server bandwidth. The async methods of controller are handled by asp.net framework so we can just implement async controller action methods.

We have covered the asynchronous process from the very beginning to the very end. And armed with an understanding of all this asynchronous nature, which is contrary to human nature, let's look at some useful tricks when working with asynchronous code.

Useful TAP Classes and Techniques

1. The static variety of the Task class.

The Task class has several useful static methods. Below are the main ones.

1. Task.WhenAny(..) — combinator that accepts IEnumerable/params of task objects and returns a task object that completes when the fastest passed task completes. That is, it allows you to wait for one of several running tasks
2. Task.WhenAll(..) — combinator that accepts IEnumerable/params of task objects and returns a task object that completes when all passed tasks have completed
3. Task.FromResult<T>(T value) — returns the same value wrapped in a completed task. Often required when implementing existing interfaces with asynchronous methods
4. Task.Delay(..) — asynchronously waits for the specified time
5. Task.Yield() — You can use await Task.Yield(); in an asynchronous method to force the method to complete asynchronously. If there is a current synchronization context, this will post the remainder of the method's execution back to that context. Also it can be used to force another thread to execute. Even if current thread is only in the beginning of his time slice, processor will take another thread to execute.

2. ConfigureAwait

Naturally the most popular «advanced» feature. .ConfigureAwait (false) means that we DO NOT care where the continuation will be performed. This method belongs to the Task class and allows us to specify whether we need to execute the continuation in the same context where the asynchronous operation was called. By default, without using this method, the context is remembered and the continuation is executed in it using the mentioned Post method. And this behavior is expected. The context is important. ConfigureAwait make sense in libraries only, in most other cases it is redundant.

Now about the problem. As they say, it is not ignorance that is terrible, but false knowledge.

Somehow I happened to observe the code of a web application, where each asynchronous call was decorated with this accelerator. It has no effect other than visual disgust. A standard ASP.NET Core web application does not have any unique contexts (unless you write them yourself, of course). Thus, the Post method is not called there anyway.

3. TaskCompletionSource<T>

A class that allows you to easily manipulate a Task object. The class has broad capabilities, but it is most useful when we want to wrap some action in a task, the end of which occurs on an event. In general, the class was created to adapt old asynchronous methods for TAP, but as we have seen, it is used not only for this. A small example of working with this class:

public static Task<string> GetSomeDataAsync()
{
    TaskCompletionSource<string> tcs = new TaskCompletionSource<string>();
    FileSystemWatcher watcher = new FileSystemWatcher
    {
        Path = Directory.GetCurrentDirectory(),
        NotifyFilter = NotifyFilters.LastAccess,
        EnableRaisingEvents = true
    };
    watcher.Changed += (o, e) => tcs.SetResult(e.FullPath);
    return tcs.Task;
}

This class creates an asynchronous wrapper to get the name of the file that was being accessed in the current folder. That is just the example, but it makes it clear, for what and how it can be used.

4. CancellationTokenSource

Allows you to cancel an asynchronous operation. The general scheme is similar to using TaskCompletionSource. First, var cts = new CancellationTokenSource () is created, which, by the way, is IDisposable, then cts.Token is passed to the asynchronous operations. Further, following some of your logic, under certain conditions, the cts.Cancel () method is called. It can also sign up for an event or whatever.

Using the CancellationToken is good practice. When writing your own asynchronous method that does some work in the background, for example, in an infinite while, you can simply insert one line into the body of the loop: cancellationToken.ThrowIfCancellationRequested () , which will throw an OperationCanceledException. This exception is treated as canceling the operation and is not stored as an exception within the task object. Also, the IsCanceled property on the Task object will be true.

5. LongRunning

It happens when you create several tasks that will run throughout the life of the service, or just for a very long time, especially when in background services. As we recall, using a thread pool is justified by the overhead of creating a thread. However, if a stream is created rarely (yes even once an hour), then these costs are leveled and you can safely create separate threads. To do this, when creating a task, you can specify a special option:

Task.Factory.StartNew(action, TaskCreationOptions.LongRunning)

ThreadPool usually is used for fast tasks. And if we will take threads for a long time, it will be similar to reducing the amount of threads in it.
Anyway, I advise you to look at all the Task.Factory.StartNew overloads, there are many ways to flexibly customize the execution of a task for specific needs.

6. Exceptions

Due to the non-deterministic nature of asynchronous code execution, the issue of exceptions is very relevant. It would be a shame if you could not catch the exception and it would be thrown in the unknown thread, killing the process. To catch an exception in one thread and raise it in another, the ExceptionDispatchInfo class was created. To catch the exception, the static method ExceptionDispatchInfo.Capture (ex) is used, which returns ExceptionDispatchInfo. A reference to this object can be passed to any thread, which will then call the Throw () method. The throw itself does NOT occur at the place where the asynchronous operation is called, but at the place where the await operator is used. As you know, await cannot be applied to void. Thus, if the context existed, the exception will be passed by Post method. Otherwise, it will be raised on a thread from the pool. And this is almost 100% hello to the crash of the application. And it is one of the reasons why we should use Task or Task<T>, but not void.

The scheduler has an TaskScheduler.UnobservedTaskException event that fires when an UnobservedTaskException is thrown. This exception is thrown during garbage collection when the GC tries to collect a task object that has an unhandled exception.

7. IAsyncEnumerable

Before C # 8 and .NET Core 3.0, it was impossible to use an iterator (yield) block in an asynchronous method, which made life difficult and forced such a method to return Task<IEnumerable<T>>, i.e. there was no way to iterate through the collection until it was fully retrieved. Now there is such an opportunity. For this, the return type should be IAsyncEnumerable<T> (or IAsyncEnumerator<T>). To traverse such a collection, use a foreach loop with the await keyword. The WithCancellation and ConfigureAwait methods can also be called on the result of the operation, specifying the CancelationToken to be used and whether to continue in the same context.

As expected, everything is done as lazily as possible. Below is an example and the output it gives.

public class Program
{
    public static async Task Main()
    {
        Stopwatch sw = new Stopwatch();
        sw.Start();
        IAsyncEnumerable<int> enumerable = AsyncYielding();
        Console.WriteLine($"Time after calling: {sw.ElapsedMilliseconds}");

        await foreach (var element in enumerable.WithCancellation(..).ConfigureAwait(false))
        {
            Console.WriteLine($"element: {element}");
            Console.WriteLine($"Time: {sw.ElapsedMilliseconds}");
        }
    }

    static async IAsyncEnumerable<int> AsyncYielding()
    {
        foreach (var uselessElement in Enumerable.Range(1, 3))
        {
            Task task = Task.Delay(TimeSpan.FromSeconds(uselessElement));
            Console.WriteLine($"Task run: {uselessElement}");
            await task;
            yield return uselessElement;
        }
    }
}

Output:

Time after calling: 0
Task run: 1
element: 1
Time: 1033
Task run: 2
element: 2
Time: 3034
Task run: 3
element: 3
Time: 6035

8. ThreadPool

This class is actively used when programming with TAP. Therefore, I will give the minimum details of its implementation. Internally, ThreadPool has an array of queues: one for each thread + one global. When a new continuation is added, the thread that initiated this continuation matters. If this is a thread from a pool, the work is put into its own queue of this thread, if it was another thread — into the global one. When a thread is selected to execute something from ThreadPool, it looks first at his local queue. If it is empty, the thread takes jobs from the global one. If it is empty, it starts stealing jobs from queues of other threads. Also, you should never rely on the order of work, because, in fact, there is no order. The number of threads in the default pool depends on many factors, including the size of the address space.

Threads in the thread pool are background threads (property isBackground = true). This kind of threads does not keep the process alive if all foreground threads have finished.

9. Task-like type

This type (structure or class) can be used as a return value from an asynchronous method. This type must be associated with a builder type using the [AsyncMethodBuilder (..)] attribute. This type must have mentioned earlier members (GetAwaiter method) to be able to apply the await keyword to it. It can be non-parameterized for methods that do not return a value and parameterized for those that do.

The builder itself is a class or structure, the skeleton of which is shown in the example below. The SetResult method has a parameter of type T for a task-like type parameterized by T. For non-parameterized types, the method has no parameters. And again, it is conventional thing, so you don't need to implement/inherit from something.

Required builder interface

class MyTaskMethodBuilder<T>
{
    public static MyTaskMethodBuilder<T> Create();

    public void Start<TStateMachine>(ref TStateMachine stateMachine)
        where TStateMachine : IAsyncStateMachine;

    public void SetStateMachine(IAsyncStateMachine stateMachine);
    public void SetException(Exception exception);
    public void SetResult(T result);

    public void AwaitOnCompleted<TAwaiter, TStateMachine>(
        ref TAwaiter awaiter, ref TStateMachine stateMachine)
        where TAwaiter : INotifyCompletion
        where TStateMachine : IAsyncStateMachine;
    public void AwaitUnsafeOnCompleted<TAwaiter, TStateMachine>(
        ref TAwaiter awaiter, ref TStateMachine stateMachine)
        where TAwaiter : ICriticalNotifyCompletion
        where TStateMachine : IAsyncStateMachine;

    public MyTask<T> Task { get; }
}

Next, the principle of operation will be described from the point of view of writing your own Task-like type. Most of this has already been described when parsing the code generated by the compiler.

The compiler uses all of these types to generate a state machine. The compiler knows which builder to use for the known types (e.g. Task). But here we specify builder by ourselves and it will be used in code generation. By the way, if the state machine is a structure, then it will be boxed when SetStateMachine is called, the builder can cache the boxed copy if necessary. The builder should call stateMachine.MoveNext in or after the Start method to start execution. After calling Start , the value of the Task property will be returned from the method. I recommend remind generated state machine move next method with all steps.

If the state machine succeeds, the SetResult method is called, otherwise SetException . If the state machine reaches logical await, the GetAwaiter() method of task-like type is executed. It returns the awaiter. If it implements the ICriticalNotifyCompletion interface and IsCompleted = false, the state machine uses builder.AwaitUnsafeOnCompleted (ref awaiter, ref stateMachine) . The AwaitUnsafeOnCompleted method should call awaiter.OnCompleted (action) , the action should be stateMachine.MoveNext when the awaiter is marked as finished. Likewise for the INotifyCompletion interface and the builder.AwaitOnCompleted method.
You can choose between INotifyCompletion and ICriticalNotifyCompletion. First interface contains only OnCompleted(Action continuation), the second is inherited from first one and contains both OnCompleted(Action continuation) and UnsafeOnCompleted(Action continuation)
Unlike OnCompleted, UnsafeOnCompleted doesn't have to propagate ExecutionContext information.

How to use this is up to you. But I advise you to think 514 times about it before applying it in production, but not for pampering. Below is an example of usage. I sketched out just a proxy for the standard builder that prints to the console which method was called and at what time. By the way, the asynchronous Main() does not want to support the custom wait type (I am sure, lots of production project was hopelessly damaged due to this Microsoft miss). Optionally, you can modify the proxy logger by using a normal logger and logging more information.

Logging proxy-task

public class Program
{
    public static void Main()
    {
        Console.WriteLine("Start");
        JustMethod().Task.Wait(); //don't repeat it. really, do not
        Console.WriteLine("Stop");
    }

    public static async LogTask JustMethod()
    {
        await DelayWrapper(1000);
    }

    public static LogTask DelayWrapper(int milliseconds) => new LogTask { Task = Task.Delay(milliseconds)};
}

[AsyncMethodBuilder(typeof(LogMethodBuilder))]
public class LogTask
{
    public Task Task { get; set; }

    public TaskAwaiter GetAwaiter() => Task.GetAwaiter();
}

public class LogMethodBuilder
{
    private AsyncTaskMethodBuilder _methodBuilder = AsyncTaskMethodBuilder.Create();
    private LogTask _task;
    
    public static LogMethodBuilder Create()
    {
        Console.WriteLine($"Method: Create; {DateTime.Now :O}");
        return new LogMethodBuilder();
    }
    
    public void Start<TStateMachine>(ref TStateMachine stateMachine) where TStateMachine : IAsyncStateMachine
    {
        Console.WriteLine($"Method: Start; {DateTime.Now :O}");
        _methodBuilder.Start(ref stateMachine);
    }
    
    public void SetStateMachine(IAsyncStateMachine stateMachine)
    {
        Console.WriteLine($"Method: SetStateMachine; {DateTime.Now :O}");
        _methodBuilder.SetStateMachine(stateMachine);
    }
    
    public void SetException(Exception exception)
    {
        Console.WriteLine($"Method: SetException; {DateTime.Now :O}");
        _methodBuilder.SetException(exception);
    }
    
    public void SetResult()
    {
        Console.WriteLine($"Method: SetResult; {DateTime.Now :O}");
        _methodBuilder.SetResult();
    }    

    public void AwaitOnCompleted<TAwaiter, TStateMachine>(ref TAwaiter awaiter, ref TStateMachine stateMachine)
        where TAwaiter : INotifyCompletion
        where TStateMachine : IAsyncStateMachine
    {
        Console.WriteLine($"Method: AwaitOnCompleted; {DateTime.Now :O}");
        _methodBuilder.AwaitOnCompleted(ref awaiter, ref stateMachine);
    }

    public void AwaitUnsafeOnCompleted<TAwaiter, TStateMachine>(ref TAwaiter awaiter, ref TStateMachine stateMachine)
        where TAwaiter : ICriticalNotifyCompletion
        where TStateMachine : IAsyncStateMachine
    {
        Console.WriteLine($"Method: AwaitUnsafeOnCompleted; {DateTime.Now :O}");
        _methodBuilder.AwaitUnsafeOnCompleted(ref awaiter, ref stateMachine);
    }

    public LogTask Task
    {
        get
        {
            Console.WriteLine($"Property: Task; {DateTime.Now :O}");
            return _task ??= new LogTask {Task = _methodBuilder.Task};
        }
        set => _task = value;
    }
}

Output:

Start
Method: Create; 2019-10-09T17:55:13.7152733+03:00
Method: Start; 2019-10-09T17:55:13.7262226+03:00
Method: AwaitUnsafeOnCompleted; 2019-10-09T17:55:13.7275206+03:00
Property: Task; 2019-10-09T17:55:13.7292005+03:00
Method: SetResult; 2019-10-09T17:55:14.7297967+03:00
Stop

Fin.
PS. You can ask questions in comments, I will try to answer asap