Fan-In Fan-Out Concurrency Pattern in Go: A Comprehensive Guide

⚠️ How to go about this series?

1. Run Every Example: Don't just read the code. Type it out, run it, and observe the behavior.
2. Experiment and Break Things: Remove sleeps and see what happens, change channel buffer sizes, modify goroutine counts.
Breaking things teaches you how they work
3. Reason About Behavior: Before running modified code, try predicting the outcome. When you see unexpected behavior, pause and think why. Challenge the explanations.
4. Build Mental Models: Each visualization represents a concept. Try drawing your own diagrams for modified code.

In our previous post, we explored the Pipeline concurrency pattern, the building blocks of Fan-In & Fan-Out concurrency patterns. You can give it a read here:

Pipeline Concurrency Pattern in Go: A Comprehensive Visual Guide

Souvik Kar Mahapatra ・ Dec 29 '24

#go #tutorial #programming #architecture

In this post we'll cover Fan-in & Fan-out Pattern and will try to visualize them. So let's gear up as we'll be hands on through out the process.

Evolution from Pipeline Pattern

The fan-in fan-out pattern is a natural evolution of the pipeline pattern. While a pipeline processes data sequentially through stages, fan-in fan-out introduces parallel processing capabilities. Let's visualize how this evolution happens:

Fan-In Fan-Out Pattern

Imagine a restaurant kitchen during busy hours. When orders come in, multiple cooks work on different dishes simultaneously (fan-out). As they complete dishes, they come together at the service counter (fan-in).

Understanding Fan-out

Fan-out is distributing work across multiple goroutines to process data in parallel. Think of it as splitting a big task into smaller pieces that can be worked on simultaneously. Here's a simple example:

func fanOut(input 

<h3>
  
  
  Understanding Fan-in
</h3>

<p>Fan-in is the opposite of fan-out - it combines multiple input channels into a single channel. It's like a funnel that collects results from all workers into one stream. Here's how we implement it:<br>
</p>
<pre class="brush:php;toolbar:false">func fanIn(inputs ...


<p>Let's put it all together with a complete example that processes numbers in parallel:<br>
</p>
<pre class="brush:php;toolbar:false">func main() {
    // Create our input channel
    input := make(chan int)

    // Start sending numbers
    go func() {
        defer close(input)
        for i := 1; i 

<h2>
  
  
  Why Use Fan-in Fan-out Pattern?
</h2>

<p><strong>Optimal Resource Utilization</strong></p>

<p>The pattern naturally distributes work across available resources, this prevents idle resources,maximizing throughput.<br>
</p>
<pre class="brush:php;toolbar:false">// Worker pool size adapts to system resources
numWorkers := runtime.NumCPU()
if numWorkers > maxWorkers {
    numWorkers = maxWorkers // Prevent over-allocation
}

Improved Performance Through Parallelization

• In the sequential approach, tasks are processed one after another, creating a linear execution time. If each task takes 1 second, processing 4 tasks takes 4 seconds.
• This parallel processing reduces total execution time to approximately (total tasks / number of workers) overhead. In our example, with 4 workers, we process all tasks in about 1.2 seconds instead of 4 seconds.

func fanOut(tasks []Task) {
    numWorkers := runtime.NumCPU() // Utilize all available CPU cores
    workers := make([]

<h2>
  
  
  Real-World Use Cases
</h2>

<p><strong>Image Processing Pipeline</strong></p>

<p>It's like a upgrade from our pipeline pattern post, we need to process faster and have dedicated go routines from each process:</p><p><img src="/static/imghw/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/173625991579012.png" class="lazy" alt="Fan-In Fan-Out Concurrency Pattern in Go: A Comprehensive Guide processing pipeline with fan in and fan out pattern" loading="lazy"    style="max-width:90%"  style="max-width:90%"></p>

<p><strong>Web Scraper Pipeline</strong><br>
Web scraping is another perfect use case for fan-in fan-out.</p>

<p><img src="/static/imghw/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/173625991836275.png" class="lazy" alt="Web scraping is another perfect use case for fan-in fan-out" loading="lazy"    style="max-width:90%"  style="max-width:90%"></p>

<p>The fan-in fan-out pattern really shines in these scenarios because it:</p>

• Manages concurrency automatically through Go's channel mechanics
• Provides natural backpressure when processing is slower than ingestion
• Allows for easy scaling by adjusting the number of workers
• Keeps the system resilient through isolated error handling

Error Handling Principles

Fail Fast: Detect and handle errors early in the pipeline

Try to perform all sort of validations before or at the start of the pipeline to make sure it doesn't fail down the line as it prevents wasting resources on invalid work that would fail later. It's especially crucial in fan-in fan-out patterns because invalid data could block workers or waste parallel processing capacity.

However it's not a hard rule and heavily depends on the business logic. Here is how we can implement it in out real-world examples:

func fanOut(input 


<p>and<br>
</p>
<pre class="brush:php;toolbar:false">func fanIn(inputs ...


<p>Notice! error in one worker the other do not stop, they keep processing and that brings us to 2nd principle</p>
<h3>
  
  
  Isolate Failures: One worker's error shouldn't affect others
</h3>

<p>In a parallel processing system, one bad task shouldn't bring down the entire system. Each worker should be independent.<br>
</p>
<pre class="brush:php;toolbar:false">func main() {
    // Create our input channel
    input := make(chan int)

    // Start sending numbers
    go func() {
        defer close(input)
        for i := 1; i 

<h4>
  
  
  Resource Cleanup: Proper cleanup on errors
</h4>

<p>Resource leaks in parallel processing can quickly escalate into system-wide issues. Proper cleanup is essential.</p>

<hr>

<p>That wraps up our deep dive into the Fan-In & Fan-Out pattern! Coming up next, we'll explore the <strong>Worker Pools concurrency pattern</strong>, which we got a glimpse of in this post. Like I said we are moving progressively clearing up dependencies before moving to the next one.</p>

<p>If you found this post helpful, have any questions, or want to share your own experiences with this pattern - I'd love to hear from you in the comments below. Your insights and questions help make these explanations even better for everyone.</p>

<p>If you missed out visual guide to Golang's goroutine and channels check it out here:</p>


<div>
  
    <div>
      <img src="/static/imghw/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/173625990185651.png" class="lazy" alt="Fan-In Fan-Out Concurrency Pattern in Go: A Comprehensive Guide" loading="lazy">
    </div>
<div>
      <h2 id="Understanding-and-visualizing-Goroutines-and-Channels-in-Golang">Understanding and visualizing Goroutines and Channels in Golang</h2>
      <h3 id="Souvik-Kar-Mahapatra-Dec">Souvik Kar Mahapatra ・ Dec 20 '24</h3>
      <div>
        #go
        #programming
        #learning
        #tutorial
      </div>
    </div>
  
</div>



<p>Stay tuned for more Go concurrency patterns! ?</p>

<p><img src="/static/imghw/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/173625992371812.gif" class="lazy" alt="Fan-In Fan-Out Concurrency Pattern in Go: A Comprehensive Guide" loading="lazy"    style="max-width:90%"  style="max-width:90%"></p>


          

            
  

            
        

The above is the detailed content of Fan-In Fan-Out Concurrency Pattern in Go: A Comprehensive Guide. For more information, please follow other related articles on the PHP Chinese website!