Solution
Approach 1: Brute Force
Algorithm
In the brute force approach, we consider every possible subarray that can be formed from the given array . For every subarray considered, we need to check whether this is the smallest unsorted subarray or not. Thus, for every such subarray considered, we find out the maximum and minimum values lying in that subarray given by and respectively.
If the subarrays and are correctly sorted, then only could be the required subrray. Further, the elements in all need to be lesser than the for satisfying the required condition. Similarly, all the elements in need to be larger than . We check for these conditions for every possible and selected.
Further, we also need to check if and are sorted correctly. If all the above conditions are satisfied, we determine the length of the unsorted subarray as . We do the same process for every subarray chosen and determine the length of the smallest unsorted subarray found.
Complexity Analysis

Time complexity : . Three nested loops are there.

Space complexity : . Constant space is used.
Approach 2: Better Brute Force
Algorithm
In this approach, we make use of an idea based on selection sort. We can traverse over the given array choosing the elements . For every such element chosen, we try to determine its correct position in the sorted array. For this, we compare with every , such that . Here, refers to the length of array.
If any happens to be lesser than , it means both and aren't at their correct position for the sorted array. Thus, we need to swap the two elements to bring them at their correct positions. Here, instead of swapping, we just note the position of (given by ) and (given by ). These two elements now mark the boundary of the unsorted subarray(atleast for the time being).
Thus, out of all the chosen, we determine the leftmost which isn't at its correct position. This marks the left boundary of the smallest unsorted subarray(). Similarly, out of all the 's considered for all 's we determine the rightmost which isn't at its correct position. This marks the right boundary of the smallest unsorted subarray().
Thus, we can determine the length of the smallest unsorted subarray as .
Complexity Analysis

Time complexity : . Two nested loops are there.

Space complexity : . Constant space is used.
Approach 3: Using Sorting
Algorithm
Another very simple idea is as follows. We can sort a copy of the given array , say given by . Then, if we compare the elements of and , we can determine the leftmost and rightmost elements which mismatch. The subarray lying between them is, then, the required shorted unsorted subarray.
Complexity Analysis

Time complexity : . Sorting takes time.

Space complexity : . We are making copy of original array.
Approach 4: Using Stack
Algorithm
The idea behind this approach is also based on selective sorting. We need to determine the correct position of the minimum and the maximum element in the unsorted subarray to determine the boundaries of the required unsorted subarray.
To do so, in this implementation, we make use of a . We traverse over the array starting from the beginning. As we go on facing elements in ascending order(a rising slope), we keep on pushing the elements' indices over the . This is done because such elements are in the correct sorted order(as it seems till now). As soon as we encounter a falling slope, i.e. an element which is smaller than the element on the top of the , we know that isn't at its correct position.
In order to determine the correct position of , we keep on popping the elemnents from the top of the until we reach the stage where the element(corresponding to the index) on the top of the is lesser than . Let's say the popping stops when the index on 's top is . Now, has found its correct position. It needs to lie at an index .
We follow the same process while traversing over the whole array, and determine the value of minimum such . This marks the left boundary of the unsorted subarray.
Similarly, to find the right boundary of the unsorted subarray, we traverse over the array backwards. This time we keep on pushing the elements if we see a falling slope. As soon as we find a rising slope, we trace forwards now and determine the larger element's correct position. We do so for the complete array and thus, determine the right boundary.
We can look at the figure below for reference. We can observe that the slopes directly indicate the relative ordering. We can also observe that the point needs to lie just after index 0 marking the left boundary and the point needs to lie just before index 7 marking the right boundary of the unsorted subarray.
Below code is inpired by @fallcreek
Complexity Analysis

Time complexity : . Stack of size is filled.

Space complexity : . Stack size grows upto .
Approach 5: Without Using Extra Space
Algorithm
The idea behind this method is that the correct position of the minimum element in the unsorted subarray helps to determine the required left boundary. Similarly, the correct position of the maximum element in the unsorted subarray helps to determine the required right boundary.
Thus, firstly we need to determine when the correctly sorted array goes wrong. We keep a track of this by observing rising slope starting from the beginning of the array. Whenever the slope falls, we know that the unsorted array has surely started. Thus, now we determine the minimum element found till the end of the array , given by .
Similarly, we scan the array in the reverse order and when the slope becomes rising instead of falling, we start looking for the maximum element till we reach the beginning of the array, given by .
Then, we traverse over and determine the correct position of and by comparing these elements with the other array elements. e.g. To determine the correct position of , we know the initial portion of is already sorted. Thus, we need to find the first element which is just larger than . Similarly, for 's position, we need to find the first element which is just smaller than searching in backwards.
We can take this figure for reference again:
We can observe that the point needs to lie just after index 0 marking the left boundary and the point needs to lie just before index 7 marking the right boundary of the unsorted subarray.
Complexity Analysis

Time complexity : . Four loops are used.

Space complexity : . Constant space is used.
Analysis written by: @vinod23