Asynchronous Streaming using Web Api for large amount of data

Need of return large amount of data or big chunk of data (from db or large file).

Two way to accomplished this:

• 1.) use of streaming mechanism so that client fetch the data as needed.
• 2.) using C# .net 5.0 async/await asynchronous execution model, to use TPL pool thread

Continuation:
We need of method return immediately to the caller stream and fetch data only when needed. this bring to us in "state machine" using "yield" keyword which runs on the server and knows what next to fetch in to output stream when client ask for more content, this process known as continuation.

public IEnumerable<DataImage>(string imageId)
{
 using (var dbReader = GetDataReader(imageId))
            {
                // Read rows asynchronously, put data into buffer and write asynchronously
                while (reader.Read())
                {
                    Thread.Sleep(1000); yield return ConvertToDataTransformObject(reader);
                }
            }
}

while the above code accomplished the state machine requirement but unfortunately, it did not achieved our goal ie: "Return immediately to the caller"

Client App to Test:

[TestMethod]
public void GetData()
{
            
 using (var client = new HttpClient())
  {
            var sw = Stopwatch.StartNew();
            var stream = client.GetStreamAsync("http://myserver:portNumber/api/ControllerName/id").Result;
            sw.Stop();
            Console.WriteLine("Elapsed time: {0}ms", sw.ElapsedMilliseconds);
 }
}

The Solution: PushStreamContent() Class: Enables scenarios where a data producer wants to write directly (either synchronously or asynchronously) using a stream

public HttpResponseMessage Get([FromUri] int imageId)
        {
            HttpResponseMessage response = Request.CreateResponse();
 
            // Create push content with a delegate that will get called when it is time to write out 
            // the response.
            response.Content = new PushStreamContent(
                async (outputStream, httpContent, transportContext) =>
                {
                    try
                    {
                        // Execute the command and get a reader
                        using (var reader = GetDataReader(imageId))
                        {
 
                            // Read rows asynchronously, put data into buffer and write asynchronously
                            while (await reader.ReadAsync())
                            {
                                var rec = ConvertToDataTransformObject(reader);
 
                                var str = await JsonConvert.SerializeObjectAsync(rec);
 
                                var buffer = UTF8Encoding.UTF8.GetBytes(str);
 
                                // Write out data to output stream
                                await outputStream.WriteAsync(buffer, 0, buffer.Length);
                            }
                        }
                    }
                    catch(HttpException ex)
                    {
                        if (ex.ErrorCode == -2147023667) // The remote host closed the connection. 
                        {
                            return;
                        }
                    }
                    finally
                    {
                        // Close output stream as we are done
                        outputStream.Close();
                    }
                });
 
            return response;
        }

If we Test out client app, the control returns immediately and rest of the server code executed, only when client reads through obtained stream. end result will get low memory usage and scalability for big chunk of data.

Reference : Andru's blog and msdn
http://weblogs.asp.net/andresv/asynchronous-streaming-in-asp-net-webapi
https://msdn.microsoft.com/en-us/library/system.net.http.pushstreamcontent(v=vs.118).aspx